PyPI page
Home page
Author:
gtyinstinct
License:
Apache 2.0
Summary:
A high-throughput and memory-efficient inference and serving engine for LLMs
Latest version:
0.0.6.post1
Required dependencies:
aiohttp
|
av
|
datasets
|
einops
|
fastapi
|
logger
|
numpy
|
openai
|
pillow
|
pydantic
|
sgl-kernel
|
torch
|
torchvision
|
tqdm
|
transformers
|
zmq
Downloads last day:
1
Downloads last week:
10
Downloads last month:
51