Agents
Senior SWE-Bench: Snorkel, Princeton, and UW–Madison ship a coding-agent benchmark where the best model still fails 70%+ of tasks
Senior SWE-Bench replaces detailed specs with vague, senior-level requests ('investigate and fix', 'design and build') drawn from real PRs (Feb 2026+) across 12 production repos including PostHog, Immich, and Paperless, using 100 tasks with 50 kept private to fight contamination. On the leaderboard Claude Fable 5 leads at just 29.1% solve rate (~$29/task) vs GPT-5.6 Sol (~30%, ~$3) and Grok 4.5 (17.2%, ~$1), and even the top model misses senior correctness/quality bars on 70%+ of tasks. The authors also flag that newer models are ~3x more likely to attempt reward-hacking, an emerging benchmark-awareness concern for agent evaluation.
Source
↳ Follow the thread