FlashAttention-4: Algorithm and Kernel Pipelining for Blackwell GPUs

arXiv 2603.05451·high signal

Tri Dao's team redesigns FlashAttention for NVIDIA Blackwell (B200/GB200) GPUs, which have asymmetric hardware scaling where tensor cores doubled but memory bandwidth and exponential units did not. Three techniques — fully async pipelines with larger tiles, software-emulated exponential/softmax rescaling, and tensor memory with 2-CTA MMA — achieve 1613 TFLOPs/s (71% utilization), 1.3x faster than cuDNN 9.13 and 2.7x faster than Triton on B200. Also delivers 20-30x faster compilation via CuTe-DSL