Separate Quantization for Prefill Lifts 1-Bit Model Accuracy 32.5 Points

Panferov and coauthors trained separate quantized weights for a model's prefill and decode phases, according to their disaggregated quantization paper.

For anyone running 1-bit quantized models locally, the stakes are concrete. The same Qwen3.8-27B checkpoint gained 32.5 points on MMLU-Pro and 35.3 points on MMMU-Pro once prefill got its own NVFP4 weights, with decode left untouched.

Decode processes one token at a time and is memory-bound, so it keeps compact low-bit weights that cut memory traffic. Prefill processes the whole prompt in one pass and is compute-bound instead, so the authors gave it a separate checkpoint quantized to NVFP4, a format built for compute rather than memory savings.

The test reused Qwen3.8-27B's already-released GGUF decode checkpoints and added the trained NVFP4 prefill checkpoint on top, without retraining decode. Streaming that extra prefill weight set from SSD in llama.cpp produced a 1.78x time-to-first-token speedup at 8K-token prompts. The paper doesn't say how that speedup holds at shorter prompts or on other SSD hardware.

Splitting quantization by phase means the accuracy loss in 1-bit local models was never mostly a decode problem. If llama.cpp or similar runtimes add dual-checkpoint loading as a standard option, this moves from research result to shipped feature.