PyPI page
Home page
Author:
Krysyuk
Summary:
Pretraining data for language models: tokenize text once into resumable, verifiable binary shards (full install: write and read). Pretraining only; not for SFT or chat data.
Latest version:
0.1.0
Required dependencies:
tokbin-core
|
tokenizers
Downloads last day:
0
Downloads last week:
6
Downloads last month:
115
7-day smoothing