Fail-to-Pass Is a Broken Test Criterion: 'Lax' Reproduction Tests Admit Wrong Patches, and Hardening Them Hits 69.4% on SWE-bench Verified
The standard fail-to-pass criterion for LLM-generated bug reproduction tests — fails on buggy code, passes on the golden fix — is shown insufficient: many F→P tests are 'lax,' reproducing the symptom while still admitting plausible-but-incorrect patches, and co-generating test and fix together creates error coupling where the in-trajectory check passes even when both are wrong. CoHarden generates the test first, then iteratively hardens test and fix against surviving mutation patches until no lax regressions remain, reaching 69.4% Resolved and 78.9% F→P on SWE-bench Verified (+9.6pp Resolved over the strongest co-generation baseline). Anyone running agent-written tests as a repair gate is currently measuring the wrong thing.
↳ Follow the thread