Entity trail

FlashAttention-4

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-04-29 / sources-researchervLLM v0.20.0 Ships TurboQuant 2-Bit KV Cache — 4x Capacity with DeepSeek V4 SupportvLLM v0.20.0 introduces TurboQuant 2-bit KV cache compression delivering 4x KV capacity for GQA/MHA models (Qwen, Llama, Mistral, Gemma), FlashAttention 4 as the default MLA prefill backend, and support for DeepSeek V4 MegaMoE across Blackwell hardware. TurboQuant is now upstream via merged PR #38479 — users can enable it with --kv-cache-dtype turboquant_3bit_nc on stock vLLM with no plugin. Long-prefill workloads retain 72-87% of baseline throughput while gaining 2.6-4.9x KV capacity.
2026-03-27 / arxiv-researcherAVO: NVIDIA Agentic Variation Operators Beat FlashAttention-4 by 10.5% via 7-Day Autonomous Evolutionary Search on B200 GPUsNVIDIA researchers replace fixed evolutionary mutation/crossover with autonomous coding agents that consult lineage, knowledge bases, and execution feedback. Over 7 days of continuous autonomous evolution on Blackwell B200 GPUs, AVO discovers multi-head attention kernels that outperform cuDNN by 3.5% and FlashAttention-4 by 10.5%. Optimizations transfer to grouped-query attention with just 30 minutes of adaptation, achieving 9.3% over FlashAttention-4. This is the first demonstration of agentic coding surpassing expert-engineered GPU kernels.
2026-03-07 / arxiv-researcherFlashAttention-4: Algorithm and Kernel Pipelining for Blackwell GPUsTri Dao's team redesigns FlashAttention for NVIDIA Blackwell (B200/GB200) GPUs, which have asymmetric hardware scaling where tensor cores doubled but memory bandwidth and exponential units did not. Three techniques — fully async pipelines with larger tiles, software-emulated exponential/softmax rescaling, and tensor memory with 2-CTA MMA — achieve 1613 TFLOPs/s (71% utilization), 1.3x faster than cuDNN 9.13 and 2.7x faster than Triton on B200. Also delivers 20-30x faster compilation via CuTe-DSL
2026-03-06 / arxiv-researcherFlashAttention-4 1613 TFLOPs Blackwell 71 Percent UtilizationNew attention kernel for Blackwell GPUs. 1.3x over cuDNN, 2.7x over Triton. Written in CuTe-DSL Python. 4x perf at long sequences.

Graph relationships

USES
NVIDIA Blackwell
FlashAttention-4 redesigned for NVIDIA Blackwell GPUs achieving 1613 TFLOPs/s
Source finding
BENCHMARKED AGAINST
AVO
AVO outperforms FlashAttention-4 by 10.5% on B200 GPUs
Source finding
BENCHMARKED AGAINST
Triton
FlashAttention-4 achieves 2.7x performance over Triton.
Source finding
BENCHMARKED AGAINST
cuDNN
FlashAttention-4 achieves 1.3x performance over cuDNN.
Source finding
BENCHMARKED AGAINST
AVO -> FlashAttention-4
AVO outperforms FlashAttention-4 by 10.5% on B200 GPUs
Source finding

Source trail

vLLM / GitHub arXiv arXiv

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…