A 26-condition study found doing nothing beat TDD for agents

Dan Luu ran 160 runs across 26 prompting conditions for a coding agent, and doing nothing beat every one of them. The result is bad news for anyone picking a testing skill by popularity. The one with 250,000 GitHub stars scored below the field average.

Agents told to write tests first wrote roughly twice as many tests, with worse coverage than agents given no instructions. They were chasing test count, since count is easy to see and correctness isn't. The formal-methods conditions, run with Verus, Lean 4, Alloy, TLA+, and Creusot, had agents proving properties that had nothing to do with the actual bugs. Differential testing did no better. Of 160 agents that tried it, 135 encoded the same bug into both sides, so the check passed while the code stayed broken.

That skill is the ECC Rust test skill, with 38,000 forks alongside the stars. Luu found its effectiveness tracked with agents choosing not to use it. The official Hegel skill, SKILL.md plus its linked Rust reference, runs past 20,000 tokens. Agents read it in 157 of 160 runs. It added 26 to 41 percent to the cost of every run, and it produced no correctness gain. Luu's own hand-written skill scored highest. He credits instructions that push the agent off its default behavior, not ones that read like a tutorial.

A skill has to change what the agent would do by default. Otherwise it's just tokens spent restating what the model already knows, per Dan Luu's agentic testing study. Luu is upfront about the limits. This is one codebase, one model family, and one task with crisp correctness criteria. He doesn't know if the result holds for something fuzzier, like a React app.