Quantization Silently Changes 14-46% of Top-1 Retrieval Results While Classification Accuracy Holds

arXiv·medium signalXBlueskyLinkedInCopy link

"The Undetected Damage of Quantization on Retrieval" (arXiv 2609.24322, 21 Sep 2026) shows a quantized model that keeps its classification accuracy still flips 14 to 46% of its top-1 retrieval results, damage aggregate ranking metrics only partly reveal. The paper ties the failure to the gap between the two highest scores: top-1 survives quantization only when that gap exceeds twice the largest rounding error. In classification the loss function pushes the correct logit away from the rest, so most inputs have a wide enough gap; in retrieval nothing separates the top document from the second, so few queries do. The gap needs no labels, predicting pre-deployment which models will break and per-input at serving time whether the quantized answer matches full precision.