Task coverage beats resampling in repeated LLM evals

Spreading a fixed eval budget across every task before resampling any of them cuts score error by 87% compared to pooling extra runs on fewer tasks, according to a paper posted September 26 that derives bounds on how much replication a repeated eval actually needs.

That's a big swing in how trustworthy a leaderboard number is for the same total compute spend. Teams comparing models on a budget can get a tighter or looser confidence interval purely from how they allocated samples, before any model result changes.

The test: 16 models, 880 LiveCodeBench tasks, five outputs per task, identical total budget under both designs. One design touches every task at least once before spending anything on a second sample. The other pools samples uniformly, which in practice means some tasks get hit five times while others get skipped. Coverage-first cut median point-estimate error (MSE) by 87.0%. Interval width came down too, median 30.6% narrower, and coverage won in 15 of the 16 model panels.

The paper doesn't say whether the bound holds outside this setup. LiveCodeBench is one benchmark family scored by pass@k, and 880 tasks is one scale. It doesn't test benchmarks with uneven task difficulty or scoring that isn't pass@k, and whether the 87% figure holds at 88 tasks or 8,800 is untested here.

If you're running repeated evals on a fixed budget, the fix costs nothing extra: touch every task once before any task gets a second sample.