PyPI page
Home page
Author:
guan404ming
Summary:
TokLens: Looking Beyond Fertility in Tokenizer Evaluation
Latest version:
0.1.2
Required dependencies:
datasets
|
numpy
|
textalloc
|
tokenizers
Optional dependencies:
matplotlib
|
pytest
|
pytest-cov
|
ruff
|
scipy
|
tiktoken
|
transformers
Downloads last day:
6
Downloads last week:
39
Downloads last month:
230