PyPI page Home page
Author:
Christian Sdunek
License:
BSD-2-Clause
Summary:
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Latest version:
0.11.0
Downloads last day:
16
Downloads last week:
83
Downloads last month:
325
7-day smoothing