Same DeepSeek weights score 20 points apart by OpenRouter host

Mohamed Moustafa ran DeepSeek V4 Flash 0731 against multiple OpenRouter providers and got GPQA Diamond scores from 75.3% to 90.2% on the same model ID, per his OpenRouter provider test. TAU-Bench spread from 58.4% to 81.3%. Same weights, same request, different host behind the scenes.

A 20-point GPQA gap is roughly the distance between model generations. Pick the wrong provider and you can downgrade your app while thinking you upgraded your model, with no error to tell you why.

The smaller failures are worse than the headline number. DigitalOcean, GMI Cloud, Mancer and Venice barely changed their output whether reasoning.effort was set to low or max, meaning you pay for thinking tokens some hosts don't spend. Vision-capable models returned HTTP 200 with "no image provided" in the body instead of an error. Null content came back with a 200 status too, so a client checking status codes alone counts it as a success. Declared quantization didn't track quality either: providers declaring fp4 scored mid-pack among fp8 hosts, and the top performers declared nothing.

Moustafa also reports the failure mode on the other side. He pinned a three-provider fallback chain with allow_fallbacks: false to stop silent quality drift. All three providers dropped out in sequence, and the app went fully down.

Simon Willison picked up the piece the same day and reached the same fix: query the /endpoints API to see which providers serve a given model ID, then pin with provider.only to the ones you've tested, per Willison's OpenRouter follow-up.

An audit of 18 benchmarks, cited alongside these findings, found every one scores a model name and never the specific route that answered. If you're running eval numbers through OpenRouter without pinning a provider, you're not measuring the model. You're measuring whichever host answered that day.