Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-25 · high
Exact-match accuracy moved just 1.5 points in the same test, the number most dashboards would read as a wash.
Why now: Most RAG pipelines that expand their index only check aggregate accuracy, not whether individual answers flipped, which this audit method catches directly.
Expanding a RAG index flipped 10.25 percent of its answers, a new audit found.
That's with the model, prompt, retrieval policy, evidence depth and generation controls all held fixed. In the same run, exact-match accuracy moved by just 1.5 points, the kind of number that reads as a wash on a dashboard while individual answers underneath were changing places.
The test comes from a preregistered study on arXiv that ran 400 Natural Questions prompts against a frozen FineWeb prefix, first at one shard, then at seven. The researchers built what they call a Snapshot Compatibility Audit. Run the same questions twice against the same index to measure ordinary repeat disagreement, then run them again against the expanded index and subtract. What's left is churn caused by the index change alone. That subtraction produced 6.44 points of excess churn on normalized exact-match and 10.25 points on blinded semantic scoring. Forty of the 400 questions flipped in a way that held up on repeat, so the new answer wasn't a fluke.
Add capacity to an index and most teams check only whether the eval score held. This audit says that's the wrong single check: a stable aggregate score can sit on top of one in ten users getting a different answer than they would have gotten with the smaller index.
Anyone running RAG in production and tracking one accuracy number per release should be sampling answer-level diffs too, not just the average.
Each link below shares sources, entities, or timing with this story.
RAGOCR competes with RAG / Shared entity: RAG / Same source domain / Earlier coverage / Tension
Linked by a graph relationship (RAGOCR competes with RAG); both cover RAG; reported by the same outlet (arxiv.org).
Shared entity: RAG / Same source domain / Shared topic / Earlier coverage / Tension
Both cover RAG; reported by the same outlet (arxiv.org); overlapping topics (accuracy, answer).
Shared entity: RAG / Same source domain / Shared topic / Earlier coverage
Both cover RAG; reported by the same outlet (arxiv.org); overlapping topics (accuracy, answer, control).
Shared entity: RAG / Same source domain / Earlier coverage / Tension / Downstream implication
Both cover RAG; reported by the same outlet (arxiv.org); earlier RAG coverage from 2026-07-30.
Shared entity: RAG / Shared topic / Earlier coverage / Tension
Both cover RAG; overlapping topics (index, point); earlier RAG coverage from 2026-08-21.
Shared entity: RAG / Same source domain / Earlier coverage / Tension
Both cover RAG; reported by the same outlet (arxiv.org); earlier RAG coverage from 2026-08-14.
Both cover RAG; reported by the same outlet (arxiv.org); earlier RAG coverage from 2026-08-10.
Both cover RAG; reported by the same outlet (arxiv.org); earlier RAG coverage from 2026-08-04.