Baseten made GLM 5.2 20% faster by quantizing more layers

Baseten found that quantizing more layers of GLM 5.2 raised throughput 20% because the errors cancel, per an August 3 Latent Space interview with Philip Kiely and Ali Taha.

It's one piece of a larger stack, speculative decoding, disaggregated prefill/decode, and KV-cache routing each add roughly another 2x, per the interview. Together they take a naive deployment from 30-50 tokens per second to 300-400, the number that decides per-token cost for anyone self-hosting an open model.

The cost self-host pitches skip

None of that infrastructure runs itself.

Baseten says every new open model release costs hours to days of requantizing to NVFP4, plus retraining a custom speculator to match it.

That's recurring labor, tied to the release calendar of open models, not a one-time setup cost.

Read the 300-400 tokens-per-second figure as a labor cost, not a hardware spec. It holds only as long as someone keeps re-tuning the stack for every new open model release.