Prior Package-Hallucination Studies Overstated Python Rates by 9.4 Points, and Adversarial Prompts Push Hallucination up 45 Points

arXiv 2608.22652·high signalXBlueskyLinkedInCopy link

The paper shows earlier evaluation methodologies misclassified standard-library modules as hallucinations, inflating the Python rate by 9.4 percentage points. Testing seven inference-time defenses (five guided decoding strategies, Self-Refine, and a RAG-based defense) across eight models in five families and four languages, RAG reduced the package hallucination rate in 18 of 32 model-language configurations, while plain greedy decoding gave the best average mitigation-utility trade-off under their new Package Utility metric. Under adversarial prompts seeded with fabricated package names, hallucination rates surged by up to 45 points, with Ruby worst affected.