Small Draft Model Predicts Agent Failure Before It Runs

A small open-weight model scores an AI agent's finished trajectory and flags failures before a single action runs, according to a paper posted to arXiv. Wired into a pre-execution veto gate on Qwen3-Coder-480B and Claude 3.5 Sonnet, the score cut execution error rate by 6 to 8 percentage points. Token cost dropped 14 to 19%.

The scoring model never touches the big agent's logits, weights, or activations, and it doesn't sample repeatedly. It reads the full trajectory once, in a single forward pass. It separates reasoning spans from action spans, then calibrates a failure score against a verifiable objective. Cross a threshold and the gate stops the action before it fires.

The technique inverts speculative decoding, which normally uses a small model to draft tokens for a big one to approve. Here the small model judges instead of drafts, and it transferred to benchmarks outside its training set without retraining.

The paper doesn't say how the veto threshold was tuned, or whether it holds outside these two models. A threshold recalibrated for a third model could give back the 14 to 19% in token savings. Or it could let the 6 to 8 points of errors back through.