AV-AIVAT cuts agent evaluation cost 74x with certified anytime-valid stopping

arXiv·medium signalXBlueskyLinkedInCopy link

Submitted 2026-08-06 (arXiv:2608.06362), AV-AIVAT combines variance-reduced outcome estimation (AIVAT) with anytime-valid confidence sequences so an agent evaluation can stop the moment evidence suffices, without breaking statistical validity. Across 15 LLM agent configurations and 71,439 paired heads-up no-limit hold'em hands, AIVAT alone gave a median 54x variance reduction, and raw outcomes needed a median 74x as many hands as corrected outcomes to reach the same 95%-confidence ±1 BB stopping criterion. The reproducibility angle is the real contribution: the method hands a third party everything needed to recheck the verdict at that exact stopping time.