Forced Revision Makes Coding Agents Worse: Correctness Falls From 0.820 to 0.673 Across Two Revisions on HumanEval Repairs
'Looping Is Not Reliability' (arXiv 2607.24604, July 27) runs a sealed five-seed study over 30 HumanEval repairs producing 900 three-revision trajectories, and finds that under forced revision current correctness drops from 0.820 after one revision to 0.673 after two — even though ever-correct rises to 0.847. The agent finds the right patch and then loses it. Two common-state studies over 2,430 branches from identical frozen programs isolate the cause: stale traces harmed 34/135 correct starts versus 4/135 with current traces, a 22.2-point increase (task-cluster 95% CI [8.9, 37.0], exact Holm p=0.0337). The authors ship a typed revision contract that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts — the practical fix is checkpoint preservation, not more loop iterations.
↳ Follow the thread