OasisKV Moves the KV Cache Off HBM Entirely Using Lookahead Sparse Prefetching — 6.5–9.7x Less Cache per Request for 0.1 Points of Reasoning Accuracy

arXiv / HuggingFace Daily Papers·medium signalXBlueskyLinkedInCopy link

arXiv 2608.08097 (Aug 8) exploits the fact that decode-time attention is naturally sparse: rather than holding the full KV cache in HBM, it keeps a 2,048-token budget resident and speculatively prefetches the rest using lookahead prediction. Reported results are a 6.5–9.7x reduction in KV cache per request, 1.69x speedup on reasoning workloads, up to 2.1x on multi-GPU long-context serving, roughly 2x throughput under prefill-decode disaggregation, and 2.2–2.6x less decode-node host memory than full KV transfer. Accuracy lands within 0.7 points of full attention overall and 0.1 points on reasoning — which, if it replicates, is the strongest current argument that long-context serving cost is a memory-hierarchy problem rather than a hardware-budget one.