ToolFailBench: a 1,000-Task Diagnostic Benchmark That Localizes Exactly How Agent Tool Use Breaks

arXiv·high signal

ToolFailBench (arXiv:2607.04686, July 6, 2026) is a 1,000-task diagnostic benchmark across finance, medicine, law, cybersecurity, and real estate that localizes where tool use breaks — labeling traces as Tool-Skip, Result-Ignore, Output-Fabrication, or Unnecessary-Tool-Use via a rule classifier plus two LLM judges aggregated by majority vote. Across 19 frontier models the best reaches only 86.33% clean tool-use rate, and models with similar aggregate scores fail in very different ways. For builders: an off-the-shelf harness to profile your agent's specific tool-use failure modes instead of relying on a single pass/fail score.