Code-enforced state pushes an AI agent's pass rate to 0.54

Zhang, Kweon and Han built four versions of the same agent setup and changed only how task state reaches the model: a raw transcript, an exact checklist, per-turn directions from a compiled state machine, and a gate that blocks any action violating the rules. The arXiv paper runs all four against a 235B agent and a 35B agent on τ²-bench's airline-support tasks.

The gate moved the 235B agent's pass^1 score from 0.39 to 0.54 on those airline tasks, the biggest gain of the four setups and bigger than the exact checklist alone.

The 35B agent barely moved. It rarely broke policy even without the gate, so there wasn't much for a blocker to catch.

The checklist result is harder to explain away. An unverified ledger the agent wrote for itself beat the accurate checklist it was handed. Getting the state right mattered less than getting the agent to commit to a claim about it, even an unchecked one.

The gate has a failure mode too. On PM-Bench's cue-driven tasks, its own judgments about state were sometimes wrong, and those wrong calls dropped the 35B agent below its plain-transcript baseline.

Across both benchmarks, a hand-coded gate only paid off where failures were cheap to decide from code and common enough to matter. The 35B agent's policy compliance was already high on the airline tasks, so there was nothing to fix. On PM-Bench, the gate's own error rate turned a safety net into a new source of mistakes.