PyPI page
Home page
Author:
None
Summary:
Scalable Data Preprocessing Tool for Training Large Language Models
Latest version:
1.2.0
Required dependencies:
absl-py
|
comment_parser
|
cosmos-xenna
|
flash-attn
|
fsspec
|
hydra-core
|
jieba
|
loguru
|
mecab-python3
|
nemo_toolkit
|
nixl-cu12
|
omegaconf
|
onnxruntime-gpu
|
openai
|
pandas
|
pyarrow
|
pynvvideocodec
|
ray
|
torch
|
torchcodec
|
transformers
|
vllm
Optional dependencies:
albumentations
|
av
|
beautifulsoup4
|
boto3
|
cudf-cu12
|
cuml-cu12
|
cvcuda_cu12
|
data-designer
|
easydict
|
einops
|
fasttext
|
ftfy
|
gpustat
|
justext
|
librosa
|
lxml
|
mwparserfromhell
|
nemo_curator
|
nvidia-cudnn-cu12
|
nvidia-dali-cuda120
|
nvidia-ml-py
|
onnx
|
onnxruntime
|
open_clip_torch
|
opencv-python
|
peft
|
pillow
|
pycld2
|
pycuda
|
pydub
|
pylibcugraph-cu12
|
pylibraft-cu12
|
pypdfium2
|
raft-dask-cu12
|
rapidsmpf-cu12
|
ray
|
resiliparse
|
s3fs
|
s5cmd
|
scikit-learn
|
scipy
|
sentence-transformers
|
sentencepiece
|
silero-vad
|
soundfile
|
timm
|
torch
|
torchaudio
|
torchvision
|
trafilatura
|
warcio
Downloads last day:
258
Downloads last week:
1,347
Downloads last month:
6,946