Recomputed Totals Catch 60% of AI Agent Failures

A basic arithmetic check catches 60% of agent failures, per a study posted to arXiv on Aug. 3. That's the harder number for builders. A trained anomaly detector needs retraining for every new model, with accuracy falling from 0.885 to 0.527 on one it hasn't seen.

The study covered 2,823 committed agent episodes across three frameworks, three local models, and Gemini 2.5 Flash. An echo-state-network ensemble with CUSUM alarms caught 71% of mid-episode failures at a 5% false-alarm budget, for an AUROC of 0.872. It ran in about 200 microseconds a step, three orders of magnitude cheaper than an LLM-judge call.

The deterministic check works differently: it recomputes a run's stated total against the tool results actually received. That alone catches 60% of failures. Add a coverage check and it hits 96%, with zero false positives in 63 trials and none across 1,825 healthy episodes.

Detection only pays off if something acts on it. Feeding alarms into a rollback-and-rerun repair loop recovered 45% of failures, against 16% for a resampling control. That lifted task success from 52% to 73% for about one extra model call.

A related study on MCP server monitoring found the same pattern. Ranking servers by drift caught only about 10% of description changes across 19,099 servers. Statistical drift signals keep losing to a direct check.