Stress-test agent planning against broken tools, extraneous tools, and unsolvable tasks — not just happy-path benchmarks

arXiv:2606.04874·medium signal

The Agent Planning Benchmark (arXiv:2606.04874, submitted June 4, 2026) provides 4,209 multimodal cases across 22 domains with settings for extraneous tools, broken tools, and unsolvable tasks; across 12 MLLMs it exposed systematic weakness in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. The key gap is calibrated refusal — knowing when to declare a task infeasible. Actionable: add adversarial planning cases (a dead tool, an irrelevant tool, an impossible goal) to your agent's eval suite and score whether it correctly refuses or routes around them.