Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-07-31 · source-backed
PAIChecker audits SWE-bench Verified and finds misalignment across five patterns and eleven scenarios, a direct consequence of the construction pipeline pairing a PR with whatever issue its description references, then using the issue as problem statement and the patch as test oracle. Their multi-agent checker hits 92.12% binary accuracy on SWE-Gym and 91.67% on SWE-bench Multilingual across four backbones. Anyone quoting SWE-bench deltas of a couple points should treat 13.6% as a noise floor on the benchmark itself.
Each link below shares sources, entities, or timing with this story.
Shared entities / Same source domain / Shared topic / Earlier coverage
Both cover SWE, Verified; reported by the same outlet (arxiv.org); overlapping topics (benchmark, issue, swe-bench, verified).
Both cover SWE, Verified; reported by the same outlet (arxiv.org); overlapping topics (benchmark, binary).
Both cover SWE, Verified; reported by the same outlet (arxiv.org); overlapping topics (swe-bench, verified).
Shared entities / Same source domain / Earlier coverage / Tension
Both cover SWE, Verified; reported by the same outlet (arxiv.org); earlier SWE coverage from 2026-07-23.
Shared entities / Shared topic / Earlier coverage / Tension
Both cover SWE, Verified; overlapping topics (benchmark, swe-bench); earlier SWE coverage from 2026-02-12.
Shared entities / Same source domain / Shared topic
Both cover PAIChecker, SWE; reported by the same outlet (arxiv.org); overlapping topics (construction, verified).
Shared entities / Shared topic / Earlier coverage
Both cover SWE, Verified; overlapping topics (benchmark, swe-bench, verified); earlier SWE coverage from 2026-05-27.
Both cover SWE, Verified; overlapping topics (benchmark, swe-bench, verified); earlier SWE coverage from 2026-04-14.