Manual review found repair hallucinations in 72.7% of LLM program-repair cases, including patches that pass every available test

arXiv 2609.04909·high signalXBlueskyLinkedInCopy link

Three LLMs were evaluated on 832 Defects4J bugs with hallucination tracked across final patches and the intermediate artifacts that guide them. Only 21.0%-55.9% of generated patches passed the developer-written test suite, and manual analysis of 812 sampled repairs found repair hallucinations in 72.7% of cases, with incorrect causal localization accounting for 45.9% and incorrect repair strategy for 18.5%. More accurate intermediate artifacts correlated with successful repair but not reliably, and models routinely misidentified the triggering test case and mispredicted line coverage on branching control flow, which means asking an agent to explain its localization is a weak proxy for whether the fix is real.