PyPI page
Home page
Author:
None
License:
BSD 3-Clause License
Copyright (c) 2024 Meta
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the ...
Summary:
A package with common tokenizers in Python and C++
Latest version:
1.4.1
Required dependencies:
sentencepiece
|
tiktoken
|
tokenizers
Downloads last day:
23,885
Downloads last week:
208,491
Downloads last month:
755,395