EarlyEval halts agent benchmark runs mid-task and cuts up to 44% of input tokens while moving resolve rates by one to two points

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.02783 (2026-09-02) attacks agent evaluation cost from a different angle than benchmark distillation, which reduces task count but leaves per-task cost alone. EarlyEval trains a pair of LightGBM success and failure classifiers over behavioral, textual and reference-solution features and kills the run the moment either crosses a calibrated confidence threshold. Across SWE-bench Verified, TerminalBench and Toolathlon it eliminates 13-26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens at 89-97% prediction accuracy, perturbing per-agent resolve rates by only one to two percentage points. For anyone running a frontier model over an agentic benchmark repeatedly, this is the cheapest available lever.