Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

arXiv·medium signal

Identifies 'Shrinkage Bias,' a systematic negative rounding error in non-uniform FP4 (E2M1) formats that compounds across layers, and proposes UFP4 — uniform 4-bit training on E1M2/INT4 grids with a Random Hadamard Transform on all three GEMMs. Tested on Dense 1.5B, MoE 7.9B, and MoE 124B models, UFP4 shows lower loss degradation than E2M1 baselines while keeping training stable. Notable for anyone tracking low-precision pretraining economics.