Manual review found repair hallucinations in 72.7% of LLM program-repair cases.

Three models were evaluated on 832 Defects4J bugs with hallucination tracked across final patches and the intermediate artifacts guiding them (arXiv 2609.04909). Only 21.0% to 55.9% of generated patches passed the developer-written test suite, and manual analysis of 812 sampled repairs found hallucinations in 72.7% of cases, with incorrect causal localization at 45.9% and incorrect repair strategy at 18.5%. More accurate intermediate artifacts correlated with successful repair but not reliably, and models routinely misidentified the triggering test and mispredicted line coverage on branching control flow. Asking an agent to explain its localization is a weak proxy for whether the fix is real.