utf8-tokenizer

PyPI page
Home page
Author: None
Summary: True UTF-8 tokenizer for byte level models
Latest version: 0.8.2
Required dependencies: torch | transformers
Optional dependencies: datasets | evaluate | numba | pytest | pytest-xdist | ruff | scikit-learn | wandb

Downloads last day: 70
Downloads last week: 81
Downloads last month: 275