PyPI page
Home page
Author:
None
License:
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTI...
Summary:
Implementation of BPE-knockout, a morphologically informed post-processing step for BPE tokenisers.
Latest version:
2026.3.24
Required dependencies:
langcodes
|
language_data
|
numpy
|
tokenizers
|
tqdm
Optional dependencies:
datasets
|
fiject
|
modest_bauwenst
|
requests
|
scipy
|
tktkt
|
torch
|
transformers
Downloads last day:
62
Downloads last week:
111
Downloads last month:
190