Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-16 · high
It scores agents on reliable execution instead of raw accuracy, with gains reported across six benchmarks.
Why now: It surfaced in coverage dated July 16, pitched as an alternative to swapping models: train a controller on the rollouts you're already logging.
A lightweight controller decides when an AI agent should verify, retry, or branch into a different approach. The language model executor itself never changes, per arXiv 2607.05458.
Most agent benchmarks grade only whether the final answer was right. They don't grade whether the process that produced it holds up in production.
The setup treats agent execution as a finite-horizon Markov decision process, training the controller offline with advantage-weighted regression against terminal task-rubric rewards. That training data is rollouts most teams already have sitting in logs.
Gains show up across six domains, including an adapted version of the tau-bench retail benchmark and AgentBench DB-Bench. A second score, called the Harness Maturity Score, grades how reliably an agent executes, separate from whether any single run lands the right answer.
Agent reliability looks like a controller problem you can train from your own logs. Watch whether the six-domain gains hold up outside the paper's own benchmarks.
It surfaced in coverage dated July 16, pitched as an alternative to swapping models: train a controller on the rollouts you're already logging.
Each link below shares sources, entities, or timing with this story.
LLM uses OpenAI / Shared entity: Bench / Earlier coverage / Tension
Linked by a graph relationship (LLM uses OpenAI); both cover Bench; earlier Bench coverage from 2026-04-25.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
LLM uses OpenAI / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover LLM; earlier LLM coverage from 2026-06-19.
LLM uses OpenAI / Shared entity: Bench / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover Bench; earlier Bench coverage from 2026-04-24.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-14.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-22.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-10.