13.6% of SWE-bench Verified Instances Have Misaligned PR-Issue Pairs, and PAIChecker Detects It at 92% Accuracy

arXiv·high signalXBlueskyLinkedInCopy link

PAIChecker (arXiv 2607.28587, July 30) systematically audits SWE-bench Verified and finds 13.6% of instances exhibit PR-issue misalignment across five patterns and eleven fine-grained scenarios — a direct consequence of the standard construction pipeline that pairs a PR with whatever issue its description references, then uses the issue as the problem statement and the patch as test oracle. The proposed multi-agent checker combines pattern identification, cross-agent label synthesis, and code-level validation, reaching 92.12% binary accuracy on SWE-Gym and 91.67% on SWE-bench Multilingual across four LLM backbones. Anyone quoting SWE-bench deltas of a few points should treat this as a noise floor on the benchmark itself.