Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-23 · high
The paper finds the standard fail-to-pass test lets wrong patches pass, then fixes it with a 9.6-point resolution gain.
Why now: The paper posted to arXiv in July 2026, benchmarking CoHarden against the strongest existing test-and-fix co-generation approach on SWE-bench Verified.
Agent-written tests meant to verify bug fixes let broken patches pass as correct, according to a paper benchmarking repair systems on SWE-bench Verified.
That's the fail-to-pass check most automated repair systems use as a pass/fail gate. The test fails on buggy code, passes on the real fix, and counts as proof the bug is gone. Many of these tests turn out to be lax, per the paper. They reproduce the symptom but still pass patches that don't actually fix the underlying bug.
Worse. When a system generates the test and the fix in the same pass, the two can fail together in matching ways. The paper calls this error coupling: an in-trajectory check that says pass even though both the test and the patch are wrong.
The fix, called CoHarden, writes the test before the fix. It then repeatedly hardens both against mutation patches, alternate patches that pass the current test but shouldn't, until none survive. That gets to 78.9% correct fail-to-pass tests and 69.4% Resolved on SWE-bench Verified, 9.6 points above the strongest baseline that generates test and fix together.
Repair pipelines that score a bug fixed once the agent's own test passes are measuring the wrong thing. The paper's point is that this failure is silent by design: the test agrees with a wrong patch and nobody notices until the bug resurfaces.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entities / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover SWE, Verified; earlier SWE coverage from 2026-04-23.
LLM uses OpenAI / Shared entity: SWE / Earlier coverage / Tension
Linked by a graph relationship (LLM uses OpenAI); both cover SWE; earlier SWE coverage from 2026-04-25.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Shared entity: SWE / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover SWE; earlier SWE coverage from 2026-03-17.
Simon Willison released LLM / Shared topic / Tension
Linked by a graph relationship (Simon Willison released LLM); overlapping topics (against, code, test); pushes against this story (skeptical).
LLM uses OpenAI / Shared entity: SWE / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover SWE; earlier SWE coverage from 2026-07-20.
LLM uses OpenAI / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover LLM; earlier LLM coverage from 2026-06-19.