piece-tokenizer

PyPI page
Home page
Author: Ismantic
Summary: C++17 BPE, BBPE, and BytePiece tokenizer with bundled vocabularies
Latest version: 0.1.0


Downloads last day: 6
Downloads last week: 13
Downloads last month: 97
7-day smoothing