Stop feeding failed code back to the model — a placebo-controlled test shows blind resampling beats self-repair at 2.5–5.5x lower token cost
Self-repair is normally benchmarked against not retrying at all, which confounds the value of the feedback with the value of the extra attempt. This July 28 study runs the matched-budget comparison on MBPP+ across 1.5B/3B/7B models: blind resampling is the strongest condition below 7B and statistically tied with the best condition at 7B, while burning 2.5–5.5x fewer tokens. The mechanism is anchoring — showing the model its own failed attempt makes it reproduce a near-identical program 33–68% of the time versus 2–14% under blind resampling, and the informational content of real execution feedback added nothing over a content-free failure notice. Important caveat: this was tested only up to 7B, so treat it as a hypothesis to A/B on frontier models rather than a settled result.
↳ Follow the thread