Point a coding agent at your test suite as an auditor: 589 verified accepted-but-buggy submissions found in 20,375 that an official judge passed

arXiv 2608.01715·medium signalXBlueskyLinkedInCopy link

Rather than assuming online-judge verdicts are ground truth, this work uses an off-the-shelf coding agent to build adversarial test suites and pairs it with a certification chain that decides whether each flagged submission is genuinely buggy without the official judge: multiple independently written accepted solutions must agree on expected output for every test, brute-force solutions settle disagreements, and a per-problem validator certifies each failing input as legal. One agent surfaced 589 verified accepted-but-buggy submissions among 20,375 audited AtCoder submissions, with a union floor of 906 across five agents; each agent independently stayed within 1.7 points of official-suite coverage on the logic bugs those suites do catch. On post-cutoff Codeforces problems with no official suites, the same method beat all five reproduced baselines at every input budget — a directly reusable pattern for auditing whether your own suite's green checks mean anything.