SWE-bench Verified Abandoned by OpenAI After Audit Finds 59.4% of Hard Test Cases Fundamentally Flawed

MarkTechPost·medium signal

OpenAI's Frontier Evals team stopped reporting SWE-bench Verified scores after their audit discovered that 59.4% of the hardest test cases had fundamental flaws—and every major frontier model could reproduce gold-patch solutions verbatim from memory using only a task ID, meaning the benchmark measured training data exposure rather than coding ability. SWE-bench Pro has become the new standard, with Claude Mythos Preview leading at 77.8%, followed by Claude Opus 4.7 Adaptive (64.3%) and GPT-5.5 (58.6%). UC Berkeley also showed in April that automated scanning agents could reward-hack all eight major agent benchmarks to near-perfect scores without solving tasks.