PyPI page
Home page
Author:
PyTorch Team
License:
BSD 3-Clause License
Copyright (c) 2024 Meta
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the ...
Summary:
A package with common tokenizers in Python and C++
Latest version:
1.5.0
Required dependencies:
sentencepiece
|
tiktoken
|
tokenizers
Downloads last day:
19,207
Downloads last week:
128,450
Downloads last month:
689,746
7-day smoothing