Picking a benchmark subset by trajectory-embedding centroid beats random sampling and cuts token cost ~90%

arXiv 2609.24928·medium signalXBlueskyLinkedInCopy link

This is a second attack on the same problem as last week's IRT adaptive testing work, but with a different and simpler mechanism: group test instances by their outcome in a recent full run to preserve the historical pass/fail rate, then select each group's instances closest to the centroid of the trajectory embedding space, deterministically rather than by sampling. Across 76 configurations and three regression scenarios (same-config reruns, model and config changes, agent framework changes), the centroid method had the lowest estimation error of everything tested, reducing average error 3-11% and worst-case 4-11% over a typical draw of the strongest baseline, and 38-46% against that baseline's 95th-percentile draw. A 10% subset holds median estimation error below 5% while cutting token cost roughly 90%.