KV Cache Eviction Formalized and Proved Computationally Hard, With Decode-Time Correction Made Possible

arXiv 2608.28293·medium signalXBlueskyLinkedInCopy link

arXiv 2608.28293 (2026-08-28, cs.AI/cs.CL) gives KV eviction its first proper formalization, proves the problem is computationally hard, then shows that framing it probabilistically reduces it to expectation estimation approximable by sampling. That reframing makes decode-time correction for evicted entries feasible, a problem the literature had ignored, and reveals that existing eviction methods are zero-variance biased estimators that can be adapted to support it. In practice the probabilistic version with decode-time correction is more robust across tasks than existing eviction heuristics at competitive performance and the same compression budget.