paddo.dev: SWE-bench Pro Exposes Benchmark Fragmentation — Opus 4.5 Drops from 80.9% to 45.89%

paddo.dev·medium signal

paddo.dev reports that Opus 4.5 scores 80.9% on SWE-bench Verified but only 45.89% on the contamination-free Pro split — a 35-point gap that demonstrates vendor benchmark reporting can be structurally misleading. The article argues that as labs report the variant most favorable to their model, SWE-bench is fragmenting into incomparable subtests, making cross-vendor comparison unreliable without specifying which split was used.