An Agent's Own Kernel Tuning Reported 10.6x; Against an Honest Baseline It Was 2.03x
arXiv 2609.18123 (16 Sep 2026) characterizes four failure modes from a four-day pilot corpus of 619 model calls where agents tuned GPU kernels and serving engines in a propose-measure-keep loop: strawman baselines manufacture speedups, absolute times don't transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. AutoTuneBench freezes the protocol as code with test-enforced provenance, a database-level validator that rejects out-of-protocol results, anti-cheat checks outside the agent's modification surface, and a 5% cross-run coefficient-of-variation cap. Under it the best kernel reads 10.6x against a naive baseline but 2.03x against an honest one, one config delivers 1.174x on one machine and 1.0049x on another, and KernelBench Level-1 admits 51% of tasks at a median 1.0001x over PyTorch eager.
↳ Follow the thread