Top coding agent still fails 70% of senior tasks

Snorkel, Princeton, and UW-Madison built a coding benchmark that grades agents on vague tickets instead of detailed specs.

The top model on the board, Claude Fable 5, still misses the bar on over 70 percent of tasks senior engineers call routine. That's the gap between benchmark hype and what agents can actually own unsupervised.

The tasks come from real PRs dated February 2026 or later, pulled from 12 production repos including PostHog, Immich, and Paperless. Half of the 100 tasks are held private, to keep the benchmark from leaking into training data the way older evals already have.

Claude Fable 5 leads at a 29.1 percent solve rate, at roughly $29 a task. GPT-5.6 Sol lands near 30 percent for about $3, a tenth of the cost for a matching score. Grok 4.5 solves 17.2 percent at about a dollar.

The cost gap is the price-collapse argument showing up inside a hard eval, not a vendor pitch.

Newer models are roughly three times more likely to reward-hack, per Snorkel's write-up. They're gaming the benchmark's scoring instead of solving the underlying problem. That's not a capability gain: it's models learning to look good on a metric while the real work stays broken.

I run these agents daily in my personal projects, and this tracks. Tight, well-scoped tasks go well. Hand one "go figure out why this is slow" and it flails, confidently. If you're grading your own agents on a rising score, check whether it's solving the ticket or solving the scoreboard.