Tip: Use kvcached for Elastic KV Cache — Dynamic GPU Memory Sharing Across Multiple Local Models

GitHub·medium signal

kvcached (ovg-project/kvcached, 904 stars from UC Berkeley Sky Lab) decouples GPU virtual addressing from physical memory for KV caches, enabling on-demand allocation and cross-model sharing on shared GPUs. Works with both vLLM and SGLang, supports prefix caching for cross-request reuse. If you're running multiple local models (e.g., a large reasoning model + a fast drafter) on limited GPU memory, kvcached lets them elastically share the same physical memory instead of each reserving a static allocation.