FP4 pretraining with E5M3 block scales beats NVIDIA's own recipe and runs 21% faster.

This work pairs E2M1 payloads with unsigned E5M3 block scales whose wider range permits periodic tensor scaling, applies selective stochastic rounding only to backward gradients, drops the Hadamard transform entirely, and uses FP4 in every eligible internal linear. Pretraining a Nemotron-H 8B for nearly 190 billion tokens, the block-16 recipe finished with lower final-window training loss and lower held-out validation NLL under each method's own quantized-inference policy. An ablation removing both the Hadamard transform and the BF16 final-block exemption raised measured model-body token throughput 21.2%.