PyPI page
Home page
Author:
None
License:
MIT
Summary:
The fastest tokenizer for modern LLMs, up to 20x faster. Drop-in for transformers.AutoTokenizer, byte-exact, never quadratic.
Latest version:
0.1.2
Optional dependencies:
huggingface-hub
|
jinja2
|
numpy
|
tiktoken
|
tokenizers
|
transformers
Downloads last day:
6
Downloads last week:
53
Downloads last month:
415