Delay KV-cache compaction until the agent's next queries exist: immediate compaction hurts, delayed token eviction cuts cache 80% and keeps accuracy

arXiv 2608.00902·medium signalXBlueskyLinkedInCopy link

Agents accumulate long trajectories of reasoning, tool calls and environment feedback, so the KV cache becomes the inference bottleneck — but nearly all cache-compaction research assumes a static context where future queries are known offline, which agents never have. Comparing token eviction against attention matching across cheap proxy-query sources (boundary, repeat-prefill, delayed future-generation) on BrowseComp-Plus and WideSearch, the study finds that compacting a turn immediately often hurts performance, while delaying compaction so the agent's own subsequent queries can serve as the proxy recovers most of the gap. Token eviction proves more robust than attention matching under imperfect proxies, preserving most accuracy at 80% KV-cache reduction and sometimes improving throughput over no compaction at all — so proxy-query selection, not the compaction algorithm, is the design decision that matters.