Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in KV Cache Eviction Under Global Cap

arXiv·medium signal

Studies KV cache eviction under a shared global decode-time budget across seven policies (LRU, H2O, SnapKV, StreamingLLM, Ada-KV, QUEST, Random). Key finding: without structural protection of prompt-boundary tokens, all policies collapse to near-zero quality regardless of their scoring sophistication. Practical implication for long-context inference: simple protection of critical tokens matters more than clever eviction scoring, potentially simplifying inference optimization for practitioners.