An agent's own kernel tuning reported 10.6x; against an honest baseline it was 2.03x.

AutoTuneBench characterizes four failure modes from a four-day corpus of 619 model calls where agents tuned GPU kernels in a propose-measure-keep loop: strawman baselines manufacture speedups, absolute times don't transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. The fix freezes the protocol as code with test-enforced provenance, a database-level validator rejecting out-of-protocol results, anti-cheat checks outside the agent's modification surface, and a 5% cross-run coefficient-of-variation cap. Under it, one config delivers 1.174x on one machine and 1.0049x on another, and KernelBench Level-1 admits 51% of tasks at a median 1.0001x over PyTorch eager. Any self-optimizing agent that reports its own speedups needs a baseline it cannot touch.