PyPI page
Author:
Ayush Dattagupta
Summary:
Scalable Data Preprocessing Tool for Training Large Language Models
Latest version:
1.4.0
Required dependencies:
absl-py
|
comment-parser
|
cosmos-xenna
|
datasets
|
fsspec
|
hydra-core
|
jieba
|
lmdb
|
loguru
|
mecab-python3
|
omegaconf
|
openai
|
pandas
|
pyarrow
|
ray
|
torch
|
transformers
Optional dependencies:
accelerate
|
ai-dynamo
|
aiohttp
|
av
|
beautifulsoup4
|
boto3
|
cuda-python
|
cudf-cu12
|
cudf-streaming-cu12
|
cuml-cu12
|
cvcuda-cu12
|
data-designer
|
easydict
|
einops
|
fasttext
|
ftfy
|
google-cloud-translate
|
gpustat
|
iso639-lang
|
justext
|
lance-ray
|
librosa
|
lxml
|
matplotlib
|
mwparserfromhell
|
nemo-curator
|
nemo-text-processing
|
nemo-toolkit
|
nixl-cu12
|
nvidia-cudnn-cu12
|
nvidia-dali-cuda120
|
nvidia-ml-py
|
onnx
|
onnxruntime
|
onnxruntime-gpu
|
open-clip-torch
|
opencc-python-reimplemented
|
opencv-python-headless
|
peft
|
pillow
|
pyannote-audio
|
pycld2
|
pycuda
|
pydub
|
pylance
|
pylibcugraph-cu12
|
pylibraft-cu12
|
pynvvideocodec
|
pypdfium2
|
python-magic
|
quack-kernels
|
qwen-asr
|
qwen-omni-utils
|
raft-dask-cu12
|
rapidsmpf-cu12
|
ray
|
requests
|
resiliparse
|
s3fs
|
s5cmd
|
sacrebleu
|
scipy
|
sentence-transformers
|
sentencepiece
|
silero-vad
|
soundfile
|
spacy
|
timm
|
torch
|
torchaudio
|
torchcodec
|
torchlibrosa
|
torchvision
|
trafilatura
|
transformers
|
vllm
|
warcio
|
whisperx
Downloads last day:
214
Downloads last week:
1,220
Downloads last month:
3,517
7-day smoothing