Direct-P Reaches 2.13x BF16 Attention Throughput on a GB200 by Skipping the Softmax Conversion

arXiv 2609.04105·medium signalXBlueskyLinkedInCopy link

Blackwell's FP4 tensor cores do not automatically speed up attention because softmax conversion and on-chip dependencies dominate once the matrix products shrink. Direct-P maps scores directly to FP4 probabilities for noncausal inference and hits up to 2.13x the BF16 forward throughput on an NVIDIA GB200, while a separate causal path reconstructs probabilities from saved quantized queries and keys and uses FP8 gradient operands to accelerate a full single-GPU 8B parameter update by up to 1.14x. The sharp caveat for anyone tempted to push further: matched distributed training must retain FP8 probabilities and values, because every tested MXFP4 probability/value training trajectory diverged.