Skills
Your coding agent's green tests may prove nothing: 46% of passing checks carry zero bug-discriminating information
BSG-VA replays each validation command across three code states — original buggy (B), candidate patch (S), and gold developer fix (G) — to classify whether a passing test actually distinguishes the bug. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carried no bug-discriminating information, and 23.8% of baseline rollouts closed with a patch whose entire positive evidence base was of that kind. Adding bug-contrast feedback cut evidence-inadequate closure by 7.8 points (p=0.0029); roughly a third of the gain came from the reminder alone, meaning you can capture most of it by simply instructing the agent to prove the test fails on the unfixed code.
↳ Follow the thread