kitoken

PyPI page
Home page
Author: None
License: BSD-2-Clause
Summary: Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Latest version: 0.10.1


Downloads last day: 7
Downloads last week: 17
Downloads last month: 39