An audit of 254 SWE-bench submissions says the leaderboard can't order its own top thirty

arXiv 2609.17394 audited 254 SWE-bench submissions across four splits without running a single model, just by analyzing the published per-instance results. The top two entries both resolve 396 of 500. The top ten agents share 285 successes and 51 failures, leaving 164 instances that distinguish them at all. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05.

Then the number that reframes everything: within-model scaffold ranges reach 29.8 percentage points, against an 8.8-point spread across the entire top thirty. The same model, wrapped differently, moves three times further than the distance between first place and thirtieth.

I've been picking tools off these leaderboards. Everyone has. Somebody posts the new SWE-bench Verified number, it beats the previous by 1.2 points, and that becomes the reason to switch harnesses. This paper says that 1.2 points is noise, and the thing I should have been tuning instead moves 30.

The paper ships a five-step audit protocol and the instance partition, which makes this actionable rather than just deflating. Run the partition against your own task distribution. If your codebase looks nothing like the 500 SWE-bench instances (and it doesn't, Real-SWE benchmarked coding agents on private production code and the best score was 38.8%), then the 164 discriminating instances are the only ones carrying signal even in principle, and they may carry none for you.

There's a quieter implication for anyone selling agent tooling. If the top thirty are statistically indistinguishable, the competitive ground isn't the model, it's the scaffold: context management, tool design, retry policy, verification gates. Which is where the protocol-preserving context trimming result below lives, and where AgentGuard's failure-derived guardrails live. Those are 10-to-40-point effects. Leaderboard position is a 1-point effect.

Stop switching harnesses for a leaderboard delta. Instrument the one you have.