Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-24 · high
The system never attempts the task itself; it just judges one alternative move and passes the preference back as advice the actor can ignore.
Why now: The finding surfaces on August 24, when a cheaper, weaker check already looks sufficient for catching bad agent plans without a bigger evaluator model.
A comparator model improved three different AI agents in every one of nine tested settings, without ever solving the task itself. The system, called COTA, is weaker at the underlying task than the actors it corrects, per the COTA paper.
Agent builders who want to catch a bad plan mid-task have usually needed a second model that's better at the job. COTA didn't need that edge and still helped in every setting.
COTA doesn't attempt WebShop, ALFWorld or tau^3-Retail tasks on its own. It samples one alternative to the actor's proposed next step and judges only which one leads to a better outcome. It hands that preference back as advice, and the actor stays in charge and can ignore it.
Training used pairwise comparisons. Two rollouts branch from the same prefix, one following the actor's move and one following the alternative. A label marks which rollout works out better.
The paper doesn't say by how much COTA trails the actors it corrects, only that the gap holds across all nine actor and benchmark combinations.
For builders adding a runtime check, COTA's design says the check only needs to compare two outcomes, not equal the actor's skill at the task.
Each link below shares sources, entities, or timing with this story.
SkillPyramid benchmarked against WebShop / Shared entities / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (SkillPyramid benchmarked against WebShop); both cover ALFWorld, WebShop; reported by the same outlet (arxiv.org).
Shared entities / Same source domain / Shared topic / Earlier coverage
Both cover ALFWorld, WebShop; reported by the same outlet (arxiv.org); overlapping topics (agent, alfworld).
Both cover ALFWorld, WebShop; reported by the same outlet (arxiv.org); overlapping topics (agent, alfworld).
Both cover ALFWorld, WebShop; reported by the same outlet (arxiv.org); overlapping topics (agent, alfworld).
Shared entity: ALFWorld / Same source domain / Shared topic / Earlier coverage
Both cover ALFWorld; reported by the same outlet (arxiv.org); overlapping topics (agent, alfworld, branch, continuation).
Both cover ALFWorld; reported by the same outlet (arxiv.org); overlapping topics (agent, alfworld).
Shared entity: ALFWorld / Same source domain / Earlier coverage / Tension
Both cover ALFWorld; reported by the same outlet (arxiv.org); earlier ALFWorld coverage from 2026-08-18.
Shared entity: Trained / Same source domain / Earlier coverage
Both cover Trained; reported by the same outlet (arxiv.org); earlier Trained coverage from 2026-08-17.