InferScale cuts agent memory latency up to 79%

InferScale precomputes retrieved memory facts as cached GPU keys and injects them straight into vLLM's paged attention cache, per the arXiv paper. That replaces the re-prefill step most memory-augmented agents run on every single call.

Mem0, MemGPT, and Zep all re-run that prefill step on every call, the exact tax the paper targets. Across three open-weight models on the LoCoMo benchmark, InferScale cut time-to-first-token 72-79% and pushed throughput up 3.7-4.5x under concurrent load.

The trick is in how it handles position. Chunked RoPE stores each fact's keys before rotation gets applied, so a cached fact slots into any position in a new prompt. Context-Window Encoding then recovers the cross-fact context that normally gets lost when facts are cached in isolation, the paper says.

It runs through vLLM's existing KV-connector interface, so it needs no changes to the inference engine and no fine-tuning of the underlying model.

InferScale scored 60.3% accuracy on LoCoMo against Mem0's 63.3%, and the paper doesn't say how that gap moves at higher retrieval depths.

Most agents don't need the last few points of recall precision. They need to stop stalling on every turn. What's worth watching is whether anyone closes that accuracy gap without giving back the speed.