PyPI page
Home page
Author:
DataLens Africa Research
License:
Apache-2.0
Summary:
Measure the tokenization tax on African languages across frontier LLMs
Latest version:
0.1.0
Required dependencies:
datasets
|
numpy
|
pandas
|
pyarrow
|
pydantic
|
pyyaml
|
regex
|
rich
|
tiktoken
|
tokenizers
|
transformers
|
typer
|
uniseg
Optional dependencies:
anthropic
|
google-generativeai
|
hypothesis
|
matplotlib
|
pytest
|
pytest-cov
Downloads last day:
0
Downloads last week:
3
Downloads last month:
17
7-day smoothing