One in Seven SWE-bench Verified Tasks Have Mismatched Issues

PAIChecker, a new multi-agent auditor, found that 13.6% of SWE-bench Verified instances have problem statements that don't match their solution patches, per the July 30 arXiv paper.

SWE-bench Verified is a widely cited coding benchmark, and vendors cite score gaps of a few points as evidence of progress. A 13.6% ground-truth error rate makes deltas that small impossible to trust on their own.

The misalignment traces back to how the benchmark gets built: SWE-bench Verified pairs a real pull request with whatever GitHub issue its description references. It then treats the issue text as the problem statement and the merged patch as the answer key. Nothing in that process checks whether the PR actually fixes the issue it's linked to.

PAIChecker itself is a multi-agent system that combines pattern identification, cross-agent label synthesis, and code-level validation across five misalignment patterns and eleven sub-scenarios. It hit 92.12% binary accuracy on SWE-Gym and 91.67% on SWE-bench Multilingual, tested across four different LLM backbones. That range suggests the detection method generalizes rather than overfitting to one benchmark's quirks.

It joins other new benchmark audits, including separate checks on pentest-agent decoys and COBOL-migration validation.