PyPI page
Home page
Author:
None
License:
MIT
Summary:
Compact Unicode Token Encoding via Semantic-Anchored Byte-level BPE — a code-aware tokenizer
Latest version:
1.0.2
Required dependencies:
orjson
|
pyahocorasick
|
regex
|
tokenizers
|
tqdm
|
transformers
|
xxhash
Optional dependencies:
hypothesis
|
matplotlib
|
mypy
|
pytest
|
pytest-cov
|
ruff
|
tabulate
|
tiktoken
Downloads last day:
1
Downloads last week:
202
Downloads last month:
1,479