PyPI page
Home page
Author:
Krysyuk
Summary:
Pretraining data for language models: tokenize text once into resumable, verifiable binary shards and read random windows through memmap. Pretraining only; not for SFT or chat data.
Latest version:
0.1.0
Required dependencies:
numpy
Optional dependencies:
huggingface-hub
|
torch
|
zstandard
Downloads last day:
0
Downloads last week:
3
Downloads last month:
129
7-day smoothing