The SWE-bench leaderboard is mostly self-reported, and the harness is doing the work

Here's the number that should change how you read every coding-agent ranking: 99 of 100 entries on the SWE-bench Verified leaderboard are vendor-submitted. One carries an independent verification badge. The other ninety-nine are labs grading their own homework on their own scaffolds. Digital Applied laid it out on June 16, and an arXiv paper the same week made the deeper point: benchmark scores tell you what an agent got right, never how it got there.

The spread is the story. Claude Opus 4.5, the exact same weights, scored anywhere from 50.2% to 55.4% depending purely on how the agent system managed context and tool calls. Push across model versions and harnesses and you get 51.9% on Scale's setup versus 69.2% on Anthropic's. That's a 17 to 21 point swing with the model held roughly constant. The variable isn't intelligence. It's plumbing.

I've felt this directly. I rebuilt the scaffolding around a research agent in my pipeline. Same model, no prompt-engineering tricks, just better context windowing and a tighter tool-result format, and the pass rate moved more than any model upgrade had given me in months. At the time I assumed I'd gotten lucky on a few cases. Now I think that was the actual lesson and the model swaps were the placebo.

So why do we keep treating the leaderboard like a shopping list? Because "pick the model at the top" is a one-line decision and "profile your harness" is a week of unglamorous work. The leaderboard rewards the lazy read. One lab quietly stopped reporting SWE-bench entirely, which tells you they know the number isn't load-bearing.

What to do: stop A/B testing models before you've A/B tested your scaffold. Instrument your agent's trajectories, not just its final pass/fail. Where does it waste tool calls? When does it lose the thread in a long context? The arXiv "trajectories as programs" work argues you can fingerprint and even steer those patterns, which is a far better use of an afternoon than swapping opus for mythos in a config and hoping. And when a vendor quotes you a Pro score, ask which harness produced it. If the answer is "ours," it's marketing, not measurement.