Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-08-11 · source-backed
arXiv 2608.09802, accepted at COLM 2026, audits the benchmark everyone quotes in funding decks and finds a large chunk of the reported headroom is measurement error, not model failure. Their replacement is 170 expert-curated multilingual refactoring instances across Python, Java, TypeScript, Go, C, C++, and Rust, averaging 11.4 modified files and 261.6 lines per instance. Best model resolves 41.2%. On HuggingFace at swe-bench-promax/SWE-Bench-ProMax. Every SWE-bench number you've cited in the last year needs an asterisk.
Each link below shares sources, entities, or timing with this story.
Claude Code uses Python / Shared entities / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (Claude Code uses Python); both cover Bench, Python, SWE, Their; reported by the same outlet (arxiv.org).
Codex CLI uses Rust / Shared entity: Bench / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (Codex CLI uses Rust); both cover Bench; reported by the same outlet (arxiv.org).
Grok uses Rust / Shared entities / Earlier coverage
Linked by a graph relationship (Grok uses Rust); both cover Java, Rust, SWE, TypeScript; earlier Java coverage from 2026-07-29.
Claw Code uses Rust / Shared entities / Earlier coverage
Linked by a graph relationship (Claw Code uses Rust); both cover Rust, TypeScript; earlier Rust coverage from 2026-04-13.
Grok uses Rust / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Grok uses Rust); both cover Bench, SWE; overlapping topics (benchmark, model).
Codex CLI uses Rust / Shared entity: Bench / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Codex CLI uses Rust); both cover Bench; overlapping topics (asterisk, benchmark, best).
Zerostack uses Rust / Shared entity: Rust / Earlier coverage
Linked by a graph relationship (Zerostack uses Rust); both cover Rust; earlier Rust coverage from 2026-05-17.
Claude Code uses Python / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Claude Code uses Python); both cover Bench, SWE, Verified; overlapping topics (benchmark, model).