About 39% of the benchmark variance you observe comes from your inference backend.

arXiv 2608.04714 runs a fully-crossed study: three instruction-tuned models × five inference frameworks × six benchmarks × four generation modes. Swapping between HuggingFace, vLLM, Ollama and peers significantly changes scores even under greedy decoding with zero sampling noise. Variance decomposition puts ~39% of the out-of-the-box spread on the backend itself, with the rest from sampling noise and per-framework default generation parameters, both fixable by disclosing and matching config. Divergences are larger on factual benchmarks than social-bias ones. Framework name and version are almost never reported alongside scores. Start reporting yours.