SWE-Bench ProMax Finds ~60% of Unsolved SWE-bench Verified Instances Have Flawed Tests — and Its Own 170-Task Multilingual Refactoring Set Tops Out at 41.2%
arXiv / HuggingFace Daily Papers·high signal
arXiv 2608.09802 (Aug 10, accepted at COLM 2026) audits the benchmark everyone quotes and reports that roughly 60% of the unsolved instances in SWE-bench Verified contain flawed tests, meaning a chunk of the reported headroom is measurement error rather than model failure. Their replacement, SWE-Bench ProMax, is 170 expert-curated multilingual code-refactoring instances across Python, Java, TypeScript, Go, C, C++, and Rust, averaging 11.4 modified files and 261.6 lines of code per instance — far larger than typical single-file bug fixes. The best model resolves 41.2%, and the dataset is on HuggingFace at swe-bench-promax/SWE-Bench-ProMax.