PyPI page
Home page
Author:
None
License:
BSD 3-Clause License
Copyright (c) 2024 Meta
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the ...
Summary:
A package with common tokenizers in Python and C++
Latest version:
1.2.0
Required dependencies:
sentencepiece
|
tiktoken
|
tokenizers
Downloads last day:
8,149
Downloads last week:
49,280
Downloads last month:
235,859