Agents predict their own failure well (0.8847 AUROC) but cannot tell you which collaboration protocol will fix it

arXiv 2608.14927·medium signalXBlueskyLinkedInCopy link

Across 4,181 competition math problems, this study compared direct solving, iterative self-correction, planner-executor-reviewer collaboration, and multi-agent deliberation, and found the routing question splits cleanly in two. Models reliably detect when they are about to fail - 0.8847 AUROC - but reliably fail at picking which specific protocol is worth its overhead. Practically: a simple confidence-based escalation reached 78% accuracy at 45K tokens versus a frozen router's 73.8% at 71.3K tokens, while a retrospective oracle hit 92.4%, leaving 18.5-28.9 points on the table in realistic settings - so gate escalation on self-reported confidence, but don't let the model choose the escalation shape.