46% of Passing Agent Tests Prove Nothing, Paper Finds

A new paper shows 46 percent of the tests agents use to prove a fix would pass on the buggy code too, per arXiv 2607.28871. That means a passing suite doesn't guarantee the tests ever touched the bug. In the study, 23.8 percent of baseline rollouts closed with a fix whose only positive evidence was a test like that.

The method, called BSG-VA, reruns every validation command across three code states: the original bug, the agent's patch, and the developer's real fix. A test that passes in all three states never distinguished the bug from the fix. It's a green checkmark with nothing behind it.

The fix is cheap. Adding bug-contrast feedback, showing the test fail on the unfixed code before it passes on the patch, cut evidence-inadequate closures by 7.8 points (p=0.0029). Roughly a third of that gain came from the reminder alone, no new scaffolding required.

The same week, Claude Code's Boris Cherny described pointing the tool at an empty Swift codebase, screenshot-diffing its builds against Anthropic's Electron app pixel-by-pixel, for over two weeks. A separate paper, ECLoop, blocks edits whose preconditions haven't been met and lifted SWE-bench Verified scores 4.8 to 11.8 points using fewer tokens. A third paper found agents navigate to the correct file for 92 percent of required deletions but cut the exact line only 52 percent. Menlo Ventures raised $3 billion across two funds, its largest in 50 years. Partner Matt Murphy named safe, fast deployment, not code generation, as the bottleneck left to solve.