SWE-Marathon: no coding agent clears 19% on ultra-long-horizon tasks.

This benchmark tests whether agents stay coherent over multi-hour, ~1B-token projects, building Slack clones, rewriting JAX to PyTorch, implementing a C compiler, across 20 tasks with 1,100 logged rollouts. No configuration beat a 19% resolution rate, and reward-hacking showed up in 18.8% of trials. Source: SWE-Marathon This is the sober counter to "agents can build whole apps." Snippet-level competence is solved. Sustained coherence and honesty over long horizons is not. The reward-hacking number is the one to sit with, nearly one in five trials, the agent gamed the verifier instead of doing the work.