A read-only verifier caught 61% of bad agent runs for under a cent

A study posted to arXiv isolates what actually helps AI agents finish multi-step tasks correctly, and the answer splits cleanly between plans and verification. The paper ran 265 matched test cells on the tau-squared-bench Retail benchmark, comparing agents given real task-specific plans against agents given the same plans shuffled into word salad, so the prompts were the same length but the guidance was gone.

Real plans raised oracle-verified success by 7.17 percentage points, with a 90% bootstrap interval of 1.15 to 13.36. The gain wasn't spread evenly. It concentrated in the harder tasks, where an agent has more chances to go off the rails without a map.

The more interesting number sits on the verification side. A read-only terminal verifier, one that checks the final state without touching anything, rejected 61% of episodes the oracle had already flagged as invalid. It wrongly withheld 17% of episodes that were actually correct, a real false-negative cost. But per the paper, that cheap verifier captured nearly all the false-pass benefit of running the full planning-plus-verification stack together, at a fraction of the compute cost, under a cent per episode.

That's a lopsided trade for anyone building agent pipelines right now. Planning work is expensive to generate and hard to keep current as tasks change. A terminal check that just asks "did the end state match what was supposed to happen" is cheap to write and cheap to run on every episode, not just a sample.

What the paper doesn't say is whether this holds outside Retail-style tasks with clean, checkable end states. Domains without a crisp final state to inspect, open-ended writing or research tasks, don't obviously get the same shortcut.