PyPI page
Home page
Author:
Vadim Borisov
License:
MIT
Summary:
The fastest tokenizer for modern LLMs, up to 20x faster. Drop-in for transformers.AutoTokenizer, byte-exact, never quadratic.
Latest version:
0.1.3
Optional dependencies:
huggingface-hub
|
jinja2
|
numpy
|
tiktoken
|
tokenizers
|
transformers
Downloads last day:
3
Downloads last week:
108
Downloads last month:
213
7-day smoothing