PyPI page
Home page
Author:
Nils Matteson
License:
Apache-2.0
Summary:
The fork primitive for LLM inference. Snapshot a running session — weights + KV cache + scheduler state — and hydrate it into N divergent children that skip prefill. For RL rollouts, parallel coding agents, agent branching. Supports vLLM and SGLang.
Latest version:
0.6.0
Required dependencies:
torch
Optional dependencies:
boto3
|
fastapi
|
httpx
|
langchain-core
|
langgraph
|
moto
|
pytest
|
pytest-asyncio
|
sglang
|
thaw-native
|
uvicorn
|
vllm
Downloads last day:
4
Downloads last week:
28
Downloads last month:
326