To Run or Not to Run: When Code Execution Actually Pays Off in LLM Program Repair

arXiv 2606.26978·medium signal

The 'generate-run-revise' loop — iteratively executing tests to refine patches — is standard in state-of-the-art program-repair agents, but executions are slow and expensive and their real impact is underexplored. This two-stage empirical study characterizes execution behavior and analyzes its cost-effectiveness in LLM-based repair. It helps builders decide where in an agent loop test execution earns its latency and where it's wasted.