GuardianAgentBench: agents top out at 74.8% across 580 scenarios, and stronger models fail differently than weaker ones

arXiv·medium signal

A July 23 benchmark tests LangChain, LlamaIndex, and Vectara agents across 580 scenarios in six domains covering tool use, environment interaction, and planning. Maximum accuracy was 74.8%, with two distinct failure regimes: stronger models under-utilize tools while weaker models misselect and over-call them — meaning a single mitigation strategy cannot cover both. Performance degrades as tool sets grow and turn sequences lengthen. The authors' guardrail recovered 19.9% of failures at a 0.5% false positive rate, beating system-prompt defenses, which is a concrete argument for external guardrails over prompt engineering.