Rails Open-Sourced lemans, the Ruby Agent Harness Behind Its Benchmark, and Published Numbers Showing Sonnet 5 as Anthropic's Weakest Result

Ruby on Rails·high signalXBlueskyLinkedInCopy link

The Rails core team released lemans at github.com/rails/lemans on 2026-08-24 after deciding the Ruby community should not have to run Python-based Harbor. In the same post they benchmarked four models on 63 Rails tasks: ox-alpha 52/63, Terra 49/63 at $0.20 and a 182-second median run, open-weight Qwen 3.8-27B 48/63 but at a 27-minute median that forced the timeout up from 30 to 60 minutes, and Sonnet 5 at 44/63, the weakest Anthropic score they have recorded despite better Rails API recall than Opus 4.8 (25.4% vs 15.9%). The finding that better API recall did not buy a better result is the useful one for anyone tuning a harness.