PyPI page
Home page
Author:
State Cloud Intelligent Computing Team
License:
Apache 2.0
Summary:
A high-throughput and memory-efficient inference and serving engine for LLMs
Latest version:
0.1.2
Required dependencies:
aiohttp
|
click
|
cmake
|
easydict
|
fastapi
|
filelock
|
gguf
|
importlib-metadata
|
jinja2
|
librosa
|
msgspec
|
numpy
|
openai
|
outlines
|
pandas
|
pillow
|
prometheus-client
|
protobuf
|
psutil
|
py-cpuinfo
|
pydantic
|
pytest
|
pyzmq
|
ray
|
requests
|
rsa
|
sentencepiece
|
soundfile
|
sse-starlette
|
thefuzz
|
tiktoken
|
tokenizers
|
torch
|
torch-npu
|
torchvision
|
tqdm
|
transformers
|
typing-extensions
|
uvicorn
Optional dependencies:
tensorizer
Downloads last day:
1
Downloads last week:
23
Downloads last month:
26