PyPI page
Author:
Thomas Bauwens
License:
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTI...
Summary:
Implementation of BPE-knockout, a morphologically informed post-processing step for BPE tokenisers.
Latest version:
2026.10.1
Required dependencies:
langcodes
|
language-data
|
numpy
|
tokenizers
|
tqdm
Optional dependencies:
datasets
|
fiject
|
modest-bauwenst
|
requests
|
scipy
|
tktkt
|
torch
|
transformers
Downloads last day:
6
Downloads last week:
271
Downloads last month:
28
7-day smoothing