Public story · 2026-03-15 · source-backed

SWE-Bench Pro Exposes Benchmark Inflation.

Issue

2026-03-15

Confidence

source-backed

Sources

Redaction

passed

Story

GPT-5.4 leads at 57.7% on uncontaminated Pro (1,865 multi-language tasks) vs 80%+ on contaminated Verified (500 Python-only tasks). Separately, changing the evaluation harness moves scores 22% independently of model choice. Harness selection is a hidden variable that can eclipse model selection. Morph LLM

GPT competes with Claude / Shared entities / Shared topic / What happened next
AI Benchmark Exploitation Is Now Systemic. Claude Opus Uses Git History to Game SWE-bench. 48-Point Gap Between Verified and Pro Scores.
Linked by a graph relationship (GPT competes with Claude); both cover GPT, SWE, Verified; overlapping topics (benchmark, evaluation, gpt-5).
Shared entities / Same source / Shared topic / What happened next
SWE-bench Is Contaminated. The Primary Coding Agent Benchmark Is Broken.
Both cover GPT, Morph LLM, SWE, Verified; cite the same source (Morph LLM); overlapping topics (benchmark, contaminated, gpt-5, model, task).
Claude Code benchmarked against GPT / Shared entities / Shared topic / What happened next
Tencent Leaderboard Splits Same Model's Score by 13.5 Points
Linked by a graph relationship (Claude Code benchmarked against GPT); both cover GPT, Python, SWE; overlapping topics (benchmark, harness, model, task).
Claude Code benchmarked against GPT / Shared entity: GPT / Same source domain / Shared topic / What happened next / Tension
Codex on GPT-5.5 tops Terminal-Bench 2.1 at 83.4%, but the top Claude config is export-banned.
Linked by a graph relationship (Claude Code benchmarked against GPT); both cover GPT; reported by the same outlet (morphllm.com).
Claude Code benchmarked against GPT / Shared entities / Shared topic / What happened next
Local models became a first-class feature of hosted coding agents in a 48-hour window
Linked by a graph relationship (Claude Code benchmarked against GPT); both cover Python, SWE; overlapping topics (harness, model).
OpenHands uses Python / Shared entities / Shared topic / What happened next
Qwen3.8 Max wins the agentic index and burns 64 turns doing it
Linked by a graph relationship (OpenHands uses Python); both cover GPT, Verified; overlapping topics (model, task).
Claude Code benchmarked against GPT / Shared entities / Shared topic / Earlier coverage
GPT-5.4 Launches: Computer Use, Tool Search API, 1M Context
Linked by a graph relationship (Claude Code benchmarked against GPT); both cover GPT, SWE, Verified; overlapping topics (benchmark, gpt-5, model).
GPT competes with DeepSeek / Shared entity: GPT / Same source domain / Shared topic / What happened next
OpenCode hit 172K stars and knocked Cursor off the #1 spot
Linked by a graph relationship (GPT competes with DeepSeek); both cover GPT; reported by the same outlet (morphllm.com).

Source trail

Morph LLM

Entities

Bench Pro Exposes Benchmark Inflation GPT Harness Morph LLM Python Separately SWE Verified

Provenance

Canonical issue: Ramsay Research Agent — 2026-03-15
AI generated: no
Story unit: 2026-03-15-swe-bench-pro-exposes-benchmark-inflation
Labels: source-backed, canonical briefing excerpt

Fetching from the wire…

SWE-Bench Pro Exposes Benchmark Inflation.

Story

Related stories

AI Benchmark Exploitation Is Now Systemic. Claude Opus Uses Git History to Game SWE-bench. 48-Point Gap Between Verified and Pro Scores.

SWE-bench Is Contaminated. The Primary Coding Agent Benchmark Is Broken.

Tencent Leaderboard Splits Same Model's Score by 13.5 Points

Codex on GPT-5.5 tops Terminal-Bench 2.1 at 83.4%, but the top Claude config is export-banned.

Local models became a first-class feature of hosted coding agents in a 48-hour window

Qwen3.8 Max wins the agentic index and burns 64 turns doing it

GPT-5.4 Launches: Computer Use, Tool Search API, 1M Context

OpenCode hit 172K stars and knocked Cursor off the #1 spot

Source trail

Entities

Provenance

SWE-Bench Pro Exposes Benchmark Inflation.

Story

Related stories

AI Benchmark Exploitation Is Now Systemic. Claude Opus Uses Git History to Game SWE-bench. 48-Point Gap Between Verified and Pro Scores.

SWE-bench Is Contaminated. The Primary Coding Agent Benchmark Is Broken.

Tencent Leaderboard Splits Same Model's Score by 13.5 Points

Codex on GPT-5.5 tops Terminal-Bench 2.1 at 83.4%, but the top Claude config is export-banned.

Local models became a first-class feature of hosted coding agents in a 48-hour window

Qwen3.8 Max wins the agentic index and burns 64 turns doing it

GPT-5.4 Launches: Computer Use, Tool Search API, 1M Context

OpenCode hit 172K stars and knocked Cursor off the #1 spot

Source trail

Entities

Provenance