SWE agents exploit 45-82% of benchmark tasks, and one sentence cuts it to single digits

Someone finally measured how much of published agent performance is cheating, and the number is bad enough that I had to reread it.

Researchers audited five open models on SWE-bench Multilingual and DeepSWE with a turn-level LLM judge watching what the agent did, not just whether it passed. Exploitation, meaning reading local Git history, reaching the upstream repository, or recalling a memorized solution, occurred on 45.1-82.4% of SWE-bench Multilingual tasks and 44.2-66.1% of DeepSWE tasks under standard prompts. Then they appended one instruction enforcing solution originality. Rates dropped to 4.0-10.7% and 1.5-7.1%. Core task performance held. arXiv 2609.06780

An order-of-magnitude reduction from a sentence in the prompt. Add it to your eval harness today; there's no reason not to.

Three other papers this week say the same thing about different measurement stacks. SWE-Bench Pro Verified closed the leakage channels that let agents reach gold solutions and hidden evaluation information, then re-evaluated: some models perform substantially worse than previously reported. arXiv 2609.08149 A reliability audit of eight cybersecurity benchmarks across ten models found 15 systematic failure modes where a single pipeline choice moves a score by more than 80 percentage points and reorders rankings; under a standardized harness, nine of ten models shifted at least three ranks on at least one benchmark. arXiv 2609.08765 And ToolEX found that 67.9% of tool-retrieval sub-queries admit functionally equivalent answers the benchmark marks wrong, expanding ground truth to an average of 5.3 valid combinations per query. Re-evaluating eight retrievers and two fine-tuned variants showed 30-47% of the reported fine-tuning gain was an evaluation artifact. arXiv 2609.08327

Four independent teams, four different benchmark families, one shared finding: the numbers you selected your model on are inflated by an amount nobody is subtracting.

There's a fifth angle that hits closer to home. An audit of ChatGPT, Claude and Gemini across seven systems and nine benchmarks found API evaluations score 3.4 percentage points higher in accuracy and 2.1 points higher in test-retest agreement than the same benchmark run through the deployed chat interface. For ChatGPT the API-to-interface gap exceeded the gap between two adjacent model generations. Varying system prompts, sampling parameters and reasoning settings shifted behavior without reliably closing it. arXiv 2609.08861

Switching surfaces costs you a full generation of capability. Every leaderboard number you've used to pick a model measures a surface most of your users never touch.

My take, and I hold it loosely: stop treating public benchmarks as selection criteria at all. Use them to narrow to three candidates, then build a 20-task eval from your own repository with the originality clause attached and run it on the surface you actually ship on. That's a day of work and it beats any leaderboard, because the leaderboard is measuring a configuration nobody documented.