kitoken

PyPI page
Home page
Author: None
License: BSD-2-Clause
Summary: Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Latest version: 0.11.0


Downloads last day: 2
Downloads last week: 26
Downloads last month: 2,950