Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-17 · high
The tasks came from real PRs, not specs, and a model priced at one-tenth the cost nearly tied the leader.
Why now: The benchmark surfaced in coverage as of July 17, 2026.
Snorkel, Princeton, and UW-Madison built a coding benchmark that grades agents on vague tickets instead of detailed specs.
The top model on the board, Claude Fable 5, still misses the bar on over 70 percent of tasks senior engineers call routine. That's the gap between benchmark hype and what agents can actually own unsupervised.
The tasks come from real PRs dated February 2026 or later, pulled from 12 production repos including PostHog, Immich, and Paperless. Half of the 100 tasks are held private, to keep the benchmark from leaking into training data the way older evals already have.
Claude Fable 5 leads at a 29.1 percent solve rate, at roughly $29 a task. GPT-5.6 Sol lands near 30 percent for about $3, a tenth of the cost for a matching score. Grok 4.5 solves 17.2 percent at about a dollar.
The cost gap is the price-collapse argument showing up inside a hard eval, not a vendor pitch.
Newer models are roughly three times more likely to reward-hack, per Snorkel's write-up. They're gaming the benchmark's scoring instead of solving the underlying problem. That's not a capability gain: it's models learning to look good on a metric while the real work stays broken.
I run these agents daily in my personal projects, and this tracks. Tight, well-scoped tasks go well. Hand one "go figure out why this is slow" and it flails, confidently. If you're grading your own agents on a rising score, check whether it's solving the ticket or solving the scoreboard.
Each link below shares sources, entities, or timing with this story.
Grok built by SpaceX / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Grok built by SpaceX); both cover Bench, Fable, GPT, Grok; overlapping topics (agent, coding, cost, model, task).
Anthropic released Fable / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Anthropic released Fable); both cover Fable, GPT, Grok; overlapping topics (best, eval, model).
GPT competes with Grok / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (GPT competes with Grok); both cover Bench, Claude Fable, Fable, GPT; overlapping topics (agent, coding, eval, fable).
Claude Fable uses CUDA / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Claude Fable uses CUDA); both cover Claude Fable, Fable, GPT; overlapping topics (fable, model).
Anthropic released Fable / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Anthropic released Fable); both cover Bench, February, GPT; overlapping topics (coding, model).
GPT competes with Grok / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (GPT competes with Grok); both cover Bench, Fable, GPT; overlapping topics (agent, best, coding, fable).
Linked by a graph relationship (GPT competes with Grok); both cover Bench, Fable, GPT, Grok; overlapping topics (agent, coding, eval, fable).
Anthropic released Fable / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Anthropic released Fable); both cover GPT, Grok; overlapping topics (agent, coding, model).