ParEvalLayer Shows Three Public Agent Benchmarks Reach Their Final Verdict After Only 15–25% of Tasks Run

arXiv 2608.02444·medium signalXBlueskyLinkedInCopy link

Instead of reporting a partial score, ParEvalLayer reads paired outcomes for two agent systems under a comparison policy fixed in advance and records one of four states: better by the required margin, not better, needs more evidence, or abstain. Replaying completed public benchmark data as if evaluation had stopped early, three benchmarks reach the same decision as the full run after observing just 15% to 25% of task outcomes, while others require far more. The practical takeaway for anyone publishing agent comparisons: report the decision rule and how many comparisons remain undecided, not just a truncated percentage.