PyPI page
Home page
Summary:
🛏️ Trim the vocabulary of a tokenizer (and optionally its model) down to the subset you actually need.
Latest version:
0.0.4
Required dependencies:
datasets
|
huggingface-hub
|
pydantic
|
pyyaml
|
skeletoken
|
tokenizers
|
transformers
Optional dependencies:
protobuf
|
sentencepiece
|
torch
Downloads last day:
0
Downloads last week:
1
Downloads last month:
83
7-day smoothing