Skills
Don't generate synthetic tests to prime code generation — real tests add 26.3 points, synthetic ones add 1.7 (p = .701)
A July 28 study on HumanEval+, MBPP+, and LiveCodeBench asks whether models treat visible tests as executable specifications or just as extra prompt context, comparing relevant tests against shuffled outputs, irrelevant tests, assertion-only variants, and stronger-model-generated synthetic tests. Real original tests moved Qwen3.6 on LiveCodeBench from 13.1% to 39.4%, but synthetic high-quality tests added only 1.7 points over baseline at p = .701 — statistically indistinguishable from nothing. Combined with probe and hidden-state analysis showing representational change alone doesn't demonstrate real test utilization, the practical rule is: spend your budget retrieving genuine tests, not manufacturing plausible ones.
↳ Follow the thread