Package hallucination benchmarks overstated Python by 9.4 points

Earlier benchmarks overstated how often AI models invent fake Python packages by 9.4 points, because they miscounted standard-library modules as hallucinations, a new paper finds.

That correction matters because teams sizing hallucination defenses have been working from the inflated number. The same testing found that adversarial prompts, seeded with fabricated package names, push hallucination rates up by as much as 45 points. Ruby saw the largest jump.

The team ran seven inference-time defenses on eight models across five model families and four languages in a paper posted to arXiv. Retrieval-augmented generation lowered the hallucination rate in 18 of 32 model-language pairs tested. Plain greedy decoding, no retrieval, no extra fine-tuning, gave the best average trade-off. The researchers scored each defense against a new Package Utility metric that penalizes blocking real packages along with fake ones.

RAG isn't the default fix it's been sold as. It helped in barely more than half the combinations tried, and the cheapest possible option beat it on average. The place any of these defenses earns its cost is against an attacker naming fake packages on purpose, where the fabrication rate jumps. If you're weighing RAG against simpler decoding settings to cut hallucinated dependencies, test against an adversarial prompt set before trusting either one's cooperative-mode numbers.