Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-06-20 · source-backed
A systematic negative rounding error in non-uniform FP4 (E2M1) formats compounds across layers, degrading training (arXiv:2606.20381). The proposed UFP4 recipe trains on uniform E1M2/INT4 grids with a Random Hadamard Transform on all three GEMMs, and shows lower loss degradation than E2M1 baselines while staying stable on Dense 1.5B, MoE 7.9B, and MoE 124B models. If you track the economics of low-precision pretraining, this is a concrete reason the naive 4-bit path was leaving accuracy on the table.
Each link below shares sources, entities, or timing with this story.
Shared entities / Earlier coverage / Tension
Both cover Dense, MoE; earlier Dense coverage from 2026-04-23; pushes against this story (versus).
Shared entity: MoE / Same source domain / What happened next / Tension
Both cover MoE; reported by the same outlet (arxiv.org); picks up the MoE thread on 2026-08-12.
Shared entity: MoE / Same source domain / What happened next / Downstream implication
Both cover MoE; reported by the same outlet (arxiv.org); picks up the MoE thread on 2026-07-20.
Shared entities / What happened next
Both cover FP4, INT4; picks up the FP4 thread on 2026-07-27.
Shared entities / Earlier coverage
Both cover Dense, MoE; earlier Dense coverage from 2026-04-13.
Both cover Dense, MoE; earlier Dense coverage from 2026-04-05.
Both cover Dense, MoE; earlier Dense coverage from 2026-04-02.
Shared entity: INT4 / Same source domain / What happened next
Both cover INT4; reported by the same outlet (arxiv.org); picks up the INT4 thread on 2026-08-20.