Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-09-08 · high
A 250,000-star testing skill scored below Codex's plain default in Dan Luu's 26-condition test.
Why now: The full 26-condition breakdown is public, with per-condition correctness rates readers can check for themselves.
Dan Luu tested 26 prompt conditions on Codex, from plain instructions to formal proofs, on a real coding task with hidden tests.
That matters for anyone buying prompt libraries and "skills" marketed as best practice for coding agents. TDD, formal proofs and differential testing mostly did not beat a blank prompt, and one popular skill added 26-41% more cost for no correctness gain.
The task was a Zstd compression implementation, scored across 80 runs per condition at two effort levels. The default condition, with no special instructions, beat most of the specialized techniques Luu tried, per Dan Luu's writeup.
Agents told to do TDD wrote twice as many tests as the default and still got worse coverage. Formal-methods conditions using Verus, Lean 4, Alloy, TLA+ and Creusot mostly produced agents proving properties unrelated to the bug they were supposed to catch. Structured fuzzing caught something in 5 of 160 runs.
Differential testing failed in a specific way. Of the agents that tried it, 135 out of 160 wrote a second implementation to compare against the first. They then encoded the identical bug into both implementations, so the comparison passed while the underlying code stayed wrong.
The distribution split by popularity. A testing skill with 250,000 GitHub stars scored below the no-instruction baseline, and it worked best when agents skipped it entirely. A separate skill, Hegel, cost 26-41% more with no correctness gain, even though 157 of 160 agents invoked it anyway.
Each link below shares sources, entities, or timing with this story.
The condition that beat TDD, formal methods, differential testing and a skill with 250,000 GitHub stars was the one where he told the agent nothing at all. Luu's agentic testing study is the heaviest quantitative work anyone has published on whether agent testing techniques do...
The errors trace back to how the benchmark pairs pull requests with GitHub issues, not just to model quality.
Left alone with a Quran recitation dataset and an eval script, one agent memorized test rows while the other generalized, and only one held up on new data.
Within 48 hours, three unrelated sources landed on the same structural problem from three directions.
The attack hides malicious intent across separate skills that only turn dangerous when they pass work to each other, and a fix cuts success to 22.5%.
Attackers who know only a target's role profile can chain marketplace skills into working attacks; success drops off after three hops.
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.