InferScale Injects Precomputed KV State Into vLLM's Paged Cache, Cutting TTFT 72–79% for Memory-Augmented Agents

arXiv 2607.27090·high signalXBlueskyLinkedInCopy link

InferScale (arXiv 2607.27090, 2026-07-29) targets the hidden tax in Mem0, MemGPT and Zep: retrieved memory facts get re-prefilled on every request. It precomputes each fact's KV representation, stores it on-GPU with a semantic embedding, and injects it directly into vLLM's paged cache via the KV-connector interface — no engine modifications, no fine-tuning — using Chunked RoPE to store keys pre-rotation and Context-Window Encoding to recover cross-fact context. Across three open-weight models on LoCoMo at k=50 it cuts TTFT by 72–79% (3.6–4.8x) and delivers 3.7–4.5x throughput under concurrent load, at 60.3% accuracy versus Mem0's 63.3%.