PyPI page
Home page
Author:
None
Summary:
Scalable Data Preprocessing Tool for Training Large Language Models
Latest version:
1.3.0
Required dependencies:
absl-py
|
ai-dynamo
|
comment_parser
|
cosmos-xenna
|
datasets
|
fsspec
|
hydra-core
|
jieba
|
lmdb
|
loguru
|
mecab-python3
|
nemo_text_processing
|
nemo_toolkit
|
nixl-cu12
|
omegaconf
|
onnxruntime-gpu
|
openai
|
pandas
|
pyannote-audio
|
pyarrow
|
pynvvideocodec
|
ray
|
torch
|
torchcodec
|
transformers
|
vllm
|
whisperx
Optional dependencies:
accelerate
|
aiohttp
|
av
|
beautifulsoup4
|
boto3
|
cuda-python
|
cudf-cu12
|
cuml-cu12
|
cvcuda_cu12
|
data-designer
|
easydict
|
einops
|
fasttext
|
ftfy
|
google-cloud-translate
|
gpustat
|
iso639-lang
|
justext
|
librosa
|
lxml
|
matplotlib
|
mwparserfromhell
|
nemo_curator
|
nvidia-cudnn-cu12
|
nvidia-dali-cuda120
|
nvidia-ml-py
|
onnx
|
onnxruntime
|
open_clip_torch
|
opencc-python-reimplemented
|
opencv-python-headless
|
peft
|
pillow
|
pycld2
|
pycuda
|
pydub
|
pylibcugraph-cu12
|
pylibraft-cu12
|
pypdfium2
|
python-magic
|
raft-dask-cu12
|
rapidsmpf-cu12
|
ray
|
requests
|
resiliparse
|
s3fs
|
s5cmd
|
sacrebleu
|
scikit-learn
|
scipy
|
sentence-transformers
|
sentencepiece
|
silero-vad
|
soundfile
|
spacy
|
timm
|
torch
|
torchaudio
|
torchvision
|
trafilatura
|
transformers
|
warcio
Downloads last day:
160
Downloads last week:
996
Downloads last month:
3,843