26-Condition Study Finds Doing Nothing Beats TDD, Formal Methods and Published Skills for Agent Correctness
Dan Luu ran 26 prompt conditions against Codex/GPT-5.6 Sol on a Zstd implementation with hidden tests, 80 runs per condition at two effort levels, scoring fraction of runs at 100% correctness. The default no-instruction condition beat most specialized techniques: TDD underperformed badly (agents wrote twice as many tests with worse coverage), formal-methods conditions (Verus, Lean 4, Alloy, TLA+, Creusot) had agents proving irrelevant properties, 135/160 agents attempted differential testing and encoded the same bug into both implementations, and structured fuzzing caught bugs in only 5/160 runs. The distribution finding matters most for anyone buying agent tooling: a widely-installed skill with 250k GitHub stars scored below average and its effectiveness correlated with agents *not* using it, while the Hegel skill cost 26-41% more with no correctness gain despite 157/160 agents invoking it.
Source
↳ Follow the thread