PyPI page
Home page
Author:
None
Summary:
A LLM serving engine extension to reduce TTFT and increase throughput, especially under long-context scenarios.
Latest version:
0.4.4
Required dependencies:
aiofile
|
aiofiles
|
aiohttp
|
awscrt
|
blake3
|
cufile-python
|
cupy-cuda12x
|
fastapi
|
httptools
|
httpx
|
msgspec
|
nixl
|
numba
|
numpy
|
nvtx
|
opentelemetry-api
|
opentelemetry-exporter-otlp
|
opentelemetry-exporter-prometheus
|
opentelemetry-sdk
|
prometheus_client
|
psutil
|
py-cpuinfo
|
pyyaml
|
pyzmq
|
redis
|
safetensors
|
setuptools
|
setuptools_scm
|
sortedcontainers
|
torch
|
transformers
|
uvicorn
Downloads last day:
2,562
Downloads last week:
12,536
Downloads last month:
112,466