Only 78 of 125 Never-Solved Terminal-Bench 3 Tasks Survive a Validity Screen, With 14 Broken Oracles and 4 Verifier Bypasses
arXiv·medium signal
arXiv 2609.26826 (20 Sep) audits a frozen Terminal-Bench 3 / Frontier-Bench 0.1 record covering 1,081 PRs, 639 scored tasks, 28,801 trials and $105,933 of agent spend. Of the 125 tasks no model passed honestly, 14 had broken oracles, 8 were dominated by infrastructure failures, 4 were passable only through verifier bypasses, and 21 could not be certified solvable. A zero pass rate on a frontier benchmark is not by itself evidence of a capability gap.