Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-08-06 · source-backed
arXiv 2608.04714 runs a fully-crossed study: three instruction-tuned models × five inference frameworks × six benchmarks × four generation modes. Swapping between HuggingFace, vLLM, Ollama and peers significantly changes scores even under greedy decoding with zero sampling noise. Variance decomposition puts ~39% of the out-of-the-box spread on the backend itself, with the rest from sampling noise and per-framework default generation parameters, both fixable by disclosing and matching config. Divergences are larger on factual benchmarks than social-bias ones. Framework name and version are almost never reported alongside scores. Start reporting yours.
Each link below shares sources, entities, or timing with this story.
Ollama supports Gemma / Shared entity: Ollama / Shared topic / Earlier coverage
Linked by a graph relationship (Ollama supports Gemma); both cover Ollama; overlapping topics (config, generation).
HuggingFace released Claude Code / Shared entity: Start / Same source domain / Earlier coverage
Linked by a graph relationship (HuggingFace released Claude Code); both cover Start; reported by the same outlet (arxiv.org).
HuggingFace released Claude Code / Shared entity: Ollama / Earlier coverage / Tension
Linked by a graph relationship (HuggingFace released Claude Code); both cover Ollama; earlier Ollama coverage from 2026-04-23.
Linked by a graph relationship (HuggingFace released Claude Code); both cover Ollama; earlier Ollama coverage from 2026-07-27.
HuggingFace released Claude Code / Shared entity: Start / Earlier coverage / Tension
Linked by a graph relationship (HuggingFace released Claude Code); both cover Start; earlier Start coverage from 2026-06-01.
Ollama supports Gemma / Shared entity: Ollama / Earlier coverage
Linked by a graph relationship (Ollama supports Gemma); both cover Ollama; earlier Ollama coverage from 2026-05-06.
HuggingFace released Claude Code / Shared entity: Framework / Earlier coverage
Linked by a graph relationship (HuggingFace released Claude Code); both cover Framework; earlier Framework coverage from 2026-05-02.
HuggingFace released Claude Code / Same source domain / Shared topic
Linked by a graph relationship (HuggingFace released Claude Code); reported by the same outlet (arxiv.org); overlapping topics (benchmark, chang).