PyPI page
Home page
Author:
UniqToken Authors
Summary:
Research tokenizer toolkit with Unigram, BPE, SuperBPE, byte fallback, and offset tracking.
Latest version:
1.0.0
Required dependencies:
regex
Optional dependencies:
accelerate
|
coverage
|
huggingface-hub
|
hypothesis
|
jinja2
|
matplotlib
|
mypy
|
numpy
|
pyarrow
|
pytest
|
ruff
|
scipy
|
sentencepiece
|
tiktoken
|
tokenizers
|
torch
|
tqdm
|
transformers
Downloads last day:
1
Downloads last week:
8
Downloads last month:
197
7-day smoothing