One paper says model IDs hide which system ran

A new paper argues that benchmark scores measure the wrong thing. IBIB audited 18 benchmarks and found none of them score the system that actually answered the request. They score a model name. The weights, the serving route, the numeric precision, and the output contract behind that name can all shift without the label changing.

The paper reports that switching only the serving arm for one declared model revision moved its score from 77.38 to 82.54. That's a swing over 5 points with the model name held constant. Separately, choosing whether to exclude failed responses from the scoring denominator changed the ranking order between systems.

Across 11 systems, IBIB ran two complete evaluations on identical weights. The runs failed different predicates of what the paper calls the binding gate, a check for whether a run's configuration is pinned to what it claims. A third run on the same weights passed. The advertised model identifier gave no signal that either failure mode existed.

Builders comparing models off a leaderboard number are often comparing serving configurations, not architectures. The paper doesn't say whether any benchmark maintainer has adopted a check like the binding gate, so for now the burden sits on whoever reads the leaderboard to ask what actually served the request.