PyPI page
Home page
Author:
None
Summary:
A LLM serving engine extension to reduce TTFT and increase throughput, especially under long-context scenarios.
Latest version:
0.5.1
Required dependencies:
aiofile
|
aiofiles
|
aiohttp
|
awscrt
|
blake3
|
cachetools
|
cufile-python
|
cupy-cuda13x
|
fastapi
|
google-api-core
|
google-cloud-bigtable
|
httptools
|
httpx
|
huggingface_hub
|
msgspec
|
nixl-cu13
|
numba
|
numpy
|
nvtx
|
opentelemetry-api
|
opentelemetry-exporter-otlp
|
opentelemetry-exporter-prometheus
|
opentelemetry-sdk
|
prometheus_client
|
psutil
|
py-cpuinfo
|
pytest
|
pyyaml
|
pyzmq
|
redis
|
safetensors
|
setuptools
|
setuptools_scm
|
sortedcontainers
|
torch
|
transformers
|
uvicorn
Downloads last day:
2,168
Downloads last week:
13,214
Downloads last month:
57,397