ovg-project/kvcached: UC Berkeley's Virtualized KV Cache Achieves 2-28× TTFT Reduction for Shared GPU Inference

GitHub·medium signalXBlueskyLinkedInCopy link

kvcached from UC Berkeley Sky Computing Lab brings OS-style virtual memory abstraction to LLM KV caches, enabling elastic GPU memory sharing across multiple models. Integrates with SGLang and vLLM, achieves 2-28× TTFT improvement when serving three Llama-3.1-8B models on an A100-80G under intermittent peak loads. Pipeline parallelism support added March 2026. Apache 2.0 licensed, 822 stars.