Best models score 70.4% on single-hop API calls but 2.4% on knowing when a tool-use policy makes a question unanswerable

arXiv 2608.12282·medium signalXBlueskyLinkedInCopy link

VAKRA (arXiv:2608.12282, Aug 12) benchmarks agents against 8,000+ executable APIs across 62 domains, verifying correctness by re-executing predicted tool calls against live endpoints rather than string-matching. Accuracy falls from 70.4% on single-hop endpoint tasks to 50-51% on compositional APIs and degrades over 50% as reasoning depth grows; policy-constrained unanswerable queries bottom out at 2.4%. The diagnostic that matters for builders: failures concentrate in language-mediated reasoning — entity disambiguation and cross-source grounding — not in tool invocation mechanics, so adding retries or better function schemas won't move these numbers.