Google TurboQuant at ICLR 2026: 6x KV Cache Memory Reduction at 3-Bit Quantization with Zero Accuracy Loss

Google Research·medium signal

Google Research presented TurboQuant at ICLR 2026, achieving 6x KV cache memory reduction by quantizing to just 3 bits without training or fine-tuning and with no accuracy loss on Gemma and Mistral benchmarks. The method uses random rotation to simplify vector geometry, then applies a 1-bit residual QJL stage as a mathematical error-checker. Community implementations in PyTorch (Triton kernels), MLX, and llama.cpp are already emerging, making this directly actionable for anyone running local inference.