ovg-project/kvcached: Virtualized Elastic KV Cache for Dynamic GPU Sharing — 822 Stars

GitHub·low signal

KVCached provides virtualized elastic KV cache management for dynamic GPU sharing across multiple LLM inference workloads. Instead of each model instance managing its own KV cache in GPU memory, KVCached pools and dynamically allocates cache across models, enabling GPU multiplexing. At 822 stars, it's early but addresses a critical infrastructure bottleneck for anyone running multiple LLM inference jobs on shared GPUs.