Agent benchmark finds no single fix for tool-use failures

GuardianAgentBench tested LangChain, LlamaIndex, and Vectara agents across 580 scenarios in six domains, and the best of them still failed one in four times. The benchmark, posted July 23, caps out at 74.8% accuracy. For teams shipping production agents that call tools, that's the ceiling, not the floor.

The benchmark found two failure regimes that run in opposite directions. Stronger models under-use their tools, sitting on information they should go fetch. Weaker models do the reverse: they misselect tools and over-call them, firing off requests they don't need.

No single mitigation covers both failure modes. A prompt tuned to stop a small model from over-calling tools does nothing for a large model that's already too conservative. The reverse holds too, which is a direct hit on the standard advice for agent reliability: write a better system prompt.

Accuracy also drops as tool sets grow and turn sequences lengthen. That's the direction most production agents are heading: more tools bolted on, longer sessions.

The one fix that worked across both regimes wasn't a prompt. An external guardrail, sitting outside the model and checking tool calls before they run, recovered 19.9% of failures at a 0.5% false positive rate. That beat every system-prompt defense tested.