PyPI page
Home page
Author:
None
License:
BSD 3-Clause License
Copyright (c) 2024 Meta
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the ...
Summary:
A package with common tokenizers in Python and C++
Latest version:
1.3.0
Required dependencies:
sentencepiece
|
tiktoken
|
tokenizers
Downloads last day:
7,960
Downloads last week:
78,302
Downloads last month:
307,032