attn-rot (Ggerganov's TurboQuant Lite) on the Cusp of Merging into llama.cpp — KV Cache Quantization Breakthrough

GitHub / r/LocalLLaMA·medium signal

PR #21038 by ggerganov implements Walsh-Hadamard transform rotation of attention activations before quantization — a simplified version of Google's TurboQuant that spreads outlier energy across all dimensions before KV cache quantization. Benchmarks show Qwen3.5-35B-A3B with attn-rot matches or beats q8 KV quality at q4/q5 bit widths, dramatically reducing VRAM requirements for long-context inference. The PR is actively being reviewed and community benchmarks show near-merge quality.