Statement coverage, branch coverage and mutation testing all detect close to zero of the hard faults in LLM-generated code

arXiv 2609.09315·high signalXBlueskyLinkedInCopy link

A study across 5 LLMs and 4 benchmarks collected 6,000+ faulty program instances from end-to-end pipelines where both code and tests were generated. Most LLM-introduced faults were trivial to catch, but the challenging ones resisted coverage-based and mutation-based criteria, and actual fault detection rates were often near zero because the generated test oracles failed to capture the faulty behaviour the test prefixes triggered. Prompt-aware oracles helped but not enough. Mutation testing only marginally outperformed plain coverage, which undercuts paying its much higher cost in this setting.