PyPI page
Author:
Thomas Bauwens
Summary:
Tokeniser toolkit: a collection of Pythonic subword tokenisers and text preprocessing tools.
Latest version:
2026.8.1
Required dependencies:
bpeasy
|
clavier-lib
|
dacite
|
datasets
|
evaluate
|
fugashi
|
ipadic
|
langcodes
|
nlpaug
|
pythainlp
|
regex
|
sentencepiece
|
tokenizers
Optional dependencies:
bpe-knockout
|
fiject
|
modest-bauwenst
|
pickybpe-bauwenst
|
sage-bauwenst
|
transformers
|
wtpsplit
Downloads last day:
0
Downloads last week:
96
Downloads last month:
89
7-day smoothing