Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-25 · high
Sonnet 5 recalled Rails APIs at 25.4%, nearly double Opus 4.8's 15.9%, on Rails' new open benchmark.
Why now: The scores come from Rails' August 24 release of lemans, the first outside look at how frontier models handle a real Rails codebase.
Rails released lemans, an open-source agent harness for Ruby, on August 24, then used it to benchmark four coding agents on 63 real Rails tasks.
The core team built it because Ruby developers shouldn't need Harbor, the existing Python-based harness, to grade an agent on Rails code. The 63-task suite is the first public scorecard for how four models handle real Rails code, not generic web work.
ox-alpha scored 52 of 63 tasks, the best result among the four models tested.
Terra scored 49 of 63, at $0.20 a run and a 182-second median.
Qwen 3.8-27B, the open-weight entry, scored 48 of 63. Its median run took 27 minutes, long enough that Rails raised the task timeout from 30 minutes to 60.
Sonnet 5 scored 44 of 63, the weakest result Anthropic has recorded on the benchmark. It recalled Rails APIs more often than Opus 4.8, 25.4% against 15.9%.
The release doesn't say whether Rails plans to re-run the suite as new models ship.
Each link below shares sources, entities, or timing with this story.
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Opus built by Anthropic); both cover Anthropic, August, Opus, Sonnet; overlapping topics (agent, anthropic, august, sonnet).
Linked by a graph relationship (Opus built by Anthropic); both cover Anthropic, Opus, Python, Sonnet; overlapping topics (agent, anthropic).
Anthropic released Advisor tool / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Anthropic released Advisor tool); both cover Anthropic, Opus, Sonnet; overlapping topics (anthropic, better, sonnet).
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Opus built by Anthropic); both cover Anthropic, August, Opus, Sonnet; overlapping topics (agent, anthropic).
Anthropic released Claude / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Anthropic released Claude); both cover Anthropic, Opus, Sonnet; overlapping topics (against, anthropic).
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Opus built by Anthropic); both cover Anthropic, Opus, Sonnet; overlapping topics (agent, anthropic).
Anthropic released Claude Code / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Anthropic released Claude Code); both cover Anthropic, August, Sonnet; overlapping topics (agent, august, sonnet).
Anthropic released Claude Code / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Anthropic released Claude Code); both cover August, Qwen; overlapping topics (against, agent, august, harness).