SWE-bench Pro Private Leaderboard Reveals Harsh Generalization Drop: Claude Opus 4.1 Falls to 17.8%, GPT-5 to 14.9% on Unseen Codebases

Scale Labs·medium signal

Scale Labs' SWE-bench Pro private subset — containing previously unseen proprietary codebases — shows dramatic performance drops: Claude Opus 4.1 falls from 22.7% (public) to 17.8% resolution, and GPT-5 from 23.1% to 14.9%. This contrasts sharply with SWE-bench Verified where top models score 70%+, suggesting current coding agents may be overfitting to public repository patterns. The private leaderboard provides the most realistic measure of true generalization for coding agents in production enterprise environments with proprietary code.