Roughly 39% of the Benchmark Variance You See Out of the Box Comes From Your Inference Backend, Not the Model

arXiv 2608.04714·high signalXBlueskyLinkedInCopy link

A fully-crossed study of three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes finds that swapping between HuggingFace, vLLM, Ollama and peers significantly changes scores even under greedy, sampling-noise-free decoding — the effect is structural and strongly model-dependent. Variance decomposition attributes about 39% of the out-of-the-box spread a practitioner observes to the backend itself, with the rest coming from sampling noise and each framework's default generation parameters, both avoidable by disclosing and matching generation config. Divergences are larger on factual benchmarks than on social-bias ones, and the authors note framework names and versions are almost never reported alongside scores.