Public story · 2026-07-30 · high

InferScale cuts agent memory latency up to 79%

The gains come from skipping re-prefill entirely, but accuracy dips to 60.3% versus Mem0's 63.3%.

Why now: Covered in the July 30 briefing on agent-infrastructure research.

Issue

2026-07-30

Confidence

high

Sources

Redaction

passed

Story

InferScale precomputes retrieved memory facts as cached GPU keys and injects them straight into vLLM's paged attention cache, per the arXiv paper. That replaces the re-prefill step most memory-augmented agents run on every single call.

Mem0, MemGPT, and Zep all re-run that prefill step on every call, the exact tax the paper targets. Across three open-weight models on the LoCoMo benchmark, InferScale cut time-to-first-token 72-79% and pushed throughput up 3.7-4.5x under concurrent load.

The trick is in how it handles position. Chunked RoPE stores each fact's keys before rotation gets applied, so a cached fact slots into any position in a new prompt. Context-Window Encoding then recovers the cross-fact context that normally gets lost when facts are cached in isolation, the paper says.

It runs through vLLM's existing KV-connector interface, so it needs no changes to the inference engine and no fine-tuning of the underlying model.

InferScale scored 60.3% accuracy on LoCoMo against Mem0's 63.3%, and the paper doesn't say how that gap moves at higher retrieval depths.

Most agents don't need the last few points of recall precision. They need to stop stalling on every turn. What's worth watching is whether anyone closes that accuracy gap without giving back the speed.

Source trail

arXiv 2607.27090

Entities

Chunked RoPE Context GPU InferScale LoCoMo Mem0 MemGPT TTFT Window Encoding

Claim evidence

InferScale cuts agent memory latency up to 79%
Source

Provenance

Canonical issue: 2026-07-30
AI generated: yes
Story unit: 2026-07-30-inferscale-injects-precomputed-kv-state-into-vllm-s-paged-cache-cutting-ttft-72-79-for-memory-au
Labels: source-backed, canonical briefing excerpt

Fetching from the wire…

InferScale cuts agent memory latency up to 79%

Story

Related stories

Memori v3.3.3: Agent-Native Memory Layer. 81.95% Accuracy at 4.97% Context. 14.3K Stars.

Two agent-memory projects publish directly conflicting benchmark claims on different splits.

Unabyss and ZooData top Product Hunt back to back

EverMemOS — Agent Memory as an Operating System.

Decay agent memory on an Ebbinghaus curve instead of hoarding it.

MemPalace retires its own 100 percent memory benchmark

ByteDance's OpenViking turns agent memory into a browsable filesystem: LoCoMo 24-57% → 80-83% with 34-91% fewer input tokens.

AI Agent Memory Costs Explode Without Validated Compaction

Source trail

Entities

Claim evidence

Provenance

InferScale cuts agent memory latency up to 79%

Story

Related stories

Memori v3.3.3: Agent-Native Memory Layer. 81.95% Accuracy at 4.97% Context. 14.3K Stars.

Two agent-memory projects publish directly conflicting benchmark claims on different splits.

Unabyss and ZooData top Product Hunt back to back

EverMemOS — Agent Memory as an Operating System.

Decay agent memory on an Ebbinghaus curve instead of hoarding it.

MemPalace retires its own 100 percent memory benchmark

ByteDance's OpenViking turns agent memory into a browsable filesystem: LoCoMo 24-57% → 80-83% with 34-91% fewer input tokens.

AI Agent Memory Costs Explode Without Validated Compaction

Source trail

Entities

Claim evidence

Provenance