ovg-project/kvcached at 1K Stars: Virtualized KV Cache Achieves 2-28x TTFT Reduction by Bringing OS-Style Virtual Memory to GPU Sharing

GitHub·medium signal

kvcached decouples GPU virtual addressing from physical memory allocation for KV caches, letting multiple LLMs elastically share a single GPU without rigid memory partitioning. Instead of reserving VRAM at startup, models request virtual address space and map physical memory on demand — identical to how OS virtual memory works. Integrates with SGLang v0.4.9+ and vLLM v0.8.4+, benchmarking 2-28x time-to-first-token reduction when serving three Llama-3.1-8B models under intermittent peak loads.