Letting Each Query Choose How Many Bits of Each Key Channel to Read Cuts Million-Token Decode Time 1.67x

arXiv 2609.17652·medium signalXBlueskyLinkedInCopy link

arXiv 2609.17652 (15 Sep 2026) targets the scan that ranks all n keys for a top-k step when million-token agentic sessions keep KV caches and indexes in host memory. Fathom stores the 4-bit K cache channel-major as bit planes, so a prefix of t planes is exactly that channel's t-bit quantizer, and each query spends its bit budget by reverse water-filling over variance-weighted channel importance. At one million tokens on Qwen3-8B a decode step is 1.67x faster in GPU time than the 136-bit scans of Double Sparsity, Loki and SparQ r=32, and on real coding-agent sessions it matches the most accurate 136-bit scan's step agreement at 92 bits.