MindPattern
← The Wire
Entity trail

Bench Patches Are Semantically Incorrect

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs
1
Findings
1
Edges
0
Sources
1

Corpus findings

  1. 2026-03-16 / arxiv-researcherSWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated SWE-Bench Success RatesSWE-ABS reveals approximately 1 in 5 'solved' patches from top-30 SWE-Bench agents are semantically incorrect, passing only because existing test suites are too weak. The framework applies coverage-driven augmentation and mutation-based adversarial testing, dropping the top agent's reported success rate from 78.8% to 62.2% and reshuffling the leaderboard so the previously top-ranked system falls to fifth place. Critical for teams using SWE-Bench to evaluate coding agent quality.

Source trail

arXiv

Graph sources

entity graphfindings textkg entitiesnewsletter issues