Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-26 · high
A new paper finds a document's influence on the model's judgment fades to noise as unrelated context grows, despite accurate retrieval.
Why now: The paper posted to arXiv in August 2026, arguing retrieval accuracy alone can't certify a RAG system's judgment.
A new paper finds RAG systems keep retrieving the right document while its influence on the model's answer fades as unrelated context grows.
The stakes are for anyone building retrieval evals. A system can pass an accuracy check while its judgment has stopped depending on the document it pulled, once unrelated context grows into the tens of thousands of tokens.
The study, a paper testing RAG's retrieval-influence gap, held one piece of information fixed, a company's risk disclosure, and varied only the surrounding unrelated context, from 2,000 tokens up to 128,000 tokens. Retrieval accuracy never moved. The disclosure's effect on the model's investment judgment fell to the level of experimental noise as the context grew.
The model kept finding the document. It just stopped being swayed by it.
The researchers ran the same test on real 10-K filings, pulling disclosures out and back in, and got the same result. It held across different model families and judgment tasks, so this isn't one model's quirk.
Bigger models didn't fix it, they just bought more time. More capable models pushed the point where influence collapses further out, but the collapse still happened.
The paper also points to a fix: a short, targeted restatement of the disclosure placed next to the decision point restores its influence. Chunk-and-summarize pipelines bury the information too far from where the model has to act on it.
Benchmarks built only on retrieval hit rate will keep calling these systems reliable. Eval suites that want a real test could copy the paper's fix: restate the disclosure at the decision point and check whether the answer moves.
Each link below shares sources, entities, or timing with this story.
LLM uses OpenAI / Shared entities / Earlier coverage / Tension
Linked by a graph relationship (LLM uses OpenAI); both cover LLM, RAG; earlier LLM coverage from 2026-07-27.
Simon Willison released LLM / Shared entity: RAG / Same source domain / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover RAG; reported by the same outlet (arxiv.org).
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-16.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Same source domain / Shared topic
Linked by a graph relationship (Simon Willison released LLM); reported by the same outlet (arxiv.org); overlapping topics (capable, model).
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-21.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-17.