PyPI page
Home page
Author:
None
Summary:
A LLM serving engine extension to reduce TTFT and increase throughput, especially under long-context scenarios.
Latest version:
0.5.2.dev0
Required dependencies:
matplotlib
|
openai
|
prometheus_client
|
sortedcontainers
Downloads last day:
74
Downloads last week:
346
Downloads last month:
981