Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-04 · high
Closing the loop with rollback and a rerun lifted task success from 52% to 73% for about one extra model call, per a new study of 2,823 agent episodes.
Why now: The comparison lands as agent frameworks push more autonomy into production without any agreed way to catch a run going bad mid-episode.
A one-class echo-state-network ensemble catches 71% of mid-episode AI agent failures, per a new benchmark of 2,823 episodes across three frameworks. It does that for a fraction of an LLM judge's cost. The gap is the point: judge-based monitoring means calling an LLM every step, while the ensemble runs in about 200 microseconds per step. Cheap enough to watch every action an agent takes.
The catch is that the learned monitor doesn't transfer. Applied cold to a framework it wasn't trained on, its AUROC falls to 0.527, barely better than a coin flip. Recalibrated for that framework, it climbs to 0.885.
A simpler check skips learning entirely. It recomputes a run's stated total from the tool results the agent actually received and flags any mismatch. That catches 60% of failures, eleven points fewer than the ensemble, but produced zero false positives across 1,825 healthy episodes.
Wiring either detector into a rollback-and-rerun loop, killing a flagged episode and retrying it, pushed task success from 52% to 73%. That cost roughly one extra model call per run.
Teams will end up shipping the deterministic verifier over the fancier ensemble. Zero false positives across 1,825 healthy runs matters more in production than catching eleven extra points of failures. The verifier also doesn't need recalibrating for every new framework. The comparison lands as agent frameworks push more autonomy into production without any agreed way to catch a run going bad mid-episode.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entity: LLM / Shared topic / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; overlapping topics (actually, agent).
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-27.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-31.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-03.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-19.