Poisoned RAG context cuts Llama 3.1 accuracy to 43.5%

Accuracy for Llama 3.1 8B on a FEVER-derived fact-checking task dropped from 77.9% to 43.5% when researchers poisoned the retrieved passages feeding the model, according to a new arXiv paper that ran 588 factorial test combinations, varying zero to three of three retrieved passages.

That gap matters for anyone shipping retrieval-augmented generation in production. Swap even one of three source documents for something wrong, and the system's answers get a lot less reliable, fast.

The corruption method mattered more than the amount of it. Entity swaps, replacing a name or organization in a passage, flipped the largest share of previously correct answers. Number-based corruption barely moved accuracy while poisoned passages stayed a minority of the retrieved set, then caused a jump in errors once poisoned passages became the majority.

The paper's proxy for unsupported generation, a lexical-overlap check, fell under attack instead of rising. That means the model wasn't inventing content to paper over bad context. It was declining to answer.

The authors used coarse automated labels rather than human review, and they describe the differences between corruption strategies as suggestive, not conclusive. That's a real limit on how far these numbers generalize.

Still, the shape of the failure is useful. If a RAG system degrades under bad retrieval, check whether it's hallucinating or going quiet. Those are different problems with different fixes, one needs better retrieval filtering, the other needs a retry path that surfaces the abstention instead of treating it as a dead end.