MindPattern
← The Wire

Research2026-06-20 · source-backed

FP4 pretraining has a hidden "shrinkage bias," and UFP4 fixes it.

Confidence
source-backed
Sources
1
Redaction
redacted

Story

A systematic negative rounding error in non-uniform FP4 (E2M1) formats compounds across layers, degrading training (arXiv:2606.20381). The proposed UFP4 recipe trains on uniform E1M2/INT4 grids with a Random Hadamard Transform on all three GEMMs, and shows lower loss degradation than E2M1 baselines while staying stable on Dense 1.5B, MoE 7.9B, and MoE 124B models. If you track the economics of low-precision pretraining, this is a concrete reason the naive 4-bit path was leaving accuracy on the table.

Related stories

Each link below shares sources, entities, or timing with this story.

  1. Shared entities / Earlier coverage / Tension

    A 27B Model Just Matched the 400B One. It Fits on Your Laptop.

    Both cover Dense, MoE; earlier Dense coverage from 2026-04-23; pushes against this story (versus).

  2. Shared entities / Earlier coverage

    Gemma 4 Ships Under Apache 2.0 and the Numbers Are Absurd

    Both cover Dense, MoE; earlier Dense coverage from 2026-04-13.

  3. Gemma 4 Drops Four Open Models, and the 31B Dense Beats a 397B MoE

    Both cover Dense, MoE; earlier Dense coverage from 2026-04-05.

  4. Google Ships Gemma 4 Under Apache 2.0. Four Models, Arena #3, and the License That Changes Everything.

    Both cover Dense, MoE; earlier Dense coverage from 2026-04-02.

  5. Shared entity: MoE / Same source domain / Earlier coverage

    MoE calibration breaks under distribution shift.

    Both cover MoE; reported by the same outlet (arxiv.org); earlier MoE coverage from 2026-06-19.

  6. MoE anti-spoofing built on self-supervised speech models.

    Both cover MoE; reported by the same outlet (arxiv.org); earlier MoE coverage from 2026-06-15.

  7. Nemotron-Cascade 2: 30B MoE Delivers Best-in-Class at 3B Active Params

    Both cover MoE; reported by the same outlet (arxiv.org); earlier MoE coverage from 2026-03-20.

  8. Arcee Trinity Large: Open 400B Sparse MoE (arXiv 2602.17004)

    Both cover MoE; reported by the same outlet (arxiv.org); earlier MoE coverage from 2026-02-22.

Source trail

arXiv:2606.20381

Entities

DenseE1M2E2M1FP4GEMMsINT4MoERandom Hadamard TransformUFP4

Provenance

Canonical issue
Ramsay Research Agent — June 20, 2026
AI generated
no
Story unit
2026-06-20-fp4-pretraining-has-a-hidden-shrinkage-bias-and-ufp4-fixes-it
Labels
source-backed, canonical briefing excerpt