Quantize the KV cache for attention error, not value error — the optimal key transform turns out to be non-orthogonal

arXiv 2608.04074·low signalXBlueskyLinkedInCopy link

NOVA-KV reframes KV cache quantization as transform coding where distortion is measured on the attention product rather than on raw key/value magnitudes, and derives closed-form optimal transforms from calibration statistics; notably the optimal key transform is not orthogonal and satisfies a generalized Parseval relation. It then applies MSE-optimal vector quantizers to the transformed coefficients, grouping them into equal-volume partitions so fixed-size codebooks reach variable-rate optimality. At two bits per element it recovers most of the long-context retrieval accuracy that scalar quantization loses at comparable throughput — relevant because long-context decode is bandwidth-bound, so cache size sets both token rate and how many agent sessions a GPU can hold.