PyPI page
Author:
LMCache Team
Summary:
A LLM serving engine extension to reduce TTFT and increase throughput, especially under long-context scenarios.
Latest version:
0.5.7.post1
Required dependencies:
aiofile
|
aiofiles
|
aiohttp
|
awscrt
|
blake3
|
cachetools
|
cryptography
|
cuda-python
|
cufile-python
|
cupy-cuda13x
|
fastapi
|
google-api-core
|
google-cloud-bigtable
|
grpcio
|
grpcio-observability
|
httptools
|
httpx
|
huggingface-hub
|
msgspec
|
numba
|
numpy
|
nvtx
|
opentelemetry-api
|
opentelemetry-exporter-otlp
|
opentelemetry-exporter-prometheus
|
opentelemetry-sdk
|
prometheus-client
|
protobuf
|
psutil
|
py-cpuinfo
|
pytest
|
pyyaml
|
pyzmq
|
redis
|
safetensors
|
setuptools
|
setuptools-scm
|
sortedcontainers
|
torch
|
transformers
|
uvicorn
Optional dependencies:
confluent-kafka
|
lmcache-rust-raw-block-io
|
nixl
|
pygrpc-native
Downloads last day:
1,594
Downloads last week:
14,675
Downloads last month:
76,527
7-day smoothing