PyPI page
Home page
Author:
Zacharie Bhatti
License:
Apache-2.0
Summary:
Tokenizer analysis toolkit. Compare vocabulary coverage, compression ratios, and token boundaries across GPT-4o, Llama 3, Mistral, and any HuggingFace tokenizer.
Latest version:
0.3.0
Optional dependencies:
click
|
rich
|
sentencepiece
|
tiktoken
|
tokenizers
|
transformers
Downloads last day:
6
Downloads last week:
15
Downloads last month:
207