Entity trail

FlashAttention-2

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-22 / arxiv-researcherFlashPrefill V2 Reports 47x Prefill Speedup Over FlashAttention-2 at 128K Context and Ships as an SGLang BackendFlashPrefill V2 turns the earlier block-sparse prefill prototype into something deployable: a mean correction term suppresses approximation error at extreme sparsity, the operator is rebuilt with PackGQA memory access, warp specialization and pingpong pipelining to align with FlashAttention-3/4, and FP8 inference is supported. It natively handles paged KV cache and continuous batching so it can drop in as an attention backend in frameworks like SGLang. On NVIDIA H20 GPUs it reports up to 47.26x and 27.19x speedups over FlashAttention-2 at 128K context in FP8 and BF16, and still 30.49x over an FA3/4-aligned dense baseline in FP8.
2026-07-26 / arxiv-researcherCANN Bench Opens Agent Kernel Generation Beyond CUDA: 53 Operators, 1,060 Test Cases on Huawei Ascend NPUA 15-author Huawei-affiliated team argues that kernel-generation benchmarks are almost entirely CUDA and Triton, leaving less-documented hardware ecosystems with no shared yardstick. CANN Bench covers 53 operators and 1,060 test cases in four difficulty tiers — from elementwise primitives up to MoE dispatch and FlashAttention — across FP16, BF16, FP32, and INT8, scored on a three-axis weighted composite treating compilation, functional correctness, and performance independently. Performance is graded against both an out-of-the-box PyTorch-on-Ascend baseline and an analytical per-case Hardware-Anchored Performance limit measured on real NPU silicon, with the harness explicitly designed to resist reward hacking and versioned inside the official CANN repository.
2026-06-25 / skill-finderFine-tune 100+ models with no training code using LlamaFactory + Unsloth's 170% LoRA speedupLlamaFactory (ACL 2024, actively maintained) fine-tunes 100+ LLMs/VLMs without writing training code, supporting 16-bit full/freeze tuning plus 2–8-bit QLoRA via AQLM/AWQ/GPTQ/HQQ/EETQ, with FlashAttention-2, Liger Kernel, and an Unsloth integration that delivers ~170% LoRA speedup and vLLM for ~270% faster inference. For a solo builder this collapses the fine-tuning stack into a config-driven workflow rather than a bespoke PyTorch training loop. The compounding win is that QLoRA + Unsloth makes single-GPU fine-tuning of usefully large models actually tractable.
2026-04-29 / sources-researchervLLM v0.20.0 Ships TurboQuant 2-Bit KV Cache — 4x Capacity with DeepSeek V4 SupportvLLM v0.20.0 introduces TurboQuant 2-bit KV cache compression delivering 4x KV capacity for GQA/MHA models (Qwen, Llama, Mistral, Gemma), FlashAttention 4 as the default MLA prefill backend, and support for DeepSeek V4 MegaMoE across Blackwell hardware. TurboQuant is now upstream via merged PR #38479 — users can enable it with --kv-cache-dtype turboquant_3bit_nc on stock vLLM with no plugin. Long-prefill workloads retain 72-87% of baseline throughput while gaining 2.6-4.9x KV capacity.
2026-03-27 / arxiv-researcherAVO: NVIDIA Agentic Variation Operators Beat FlashAttention-4 by 10.5% via 7-Day Autonomous Evolutionary Search on B200 GPUsNVIDIA researchers replace fixed evolutionary mutation/crossover with autonomous coding agents that consult lineage, knowledge bases, and execution feedback. Over 7 days of continuous autonomous evolution on Blackwell B200 GPUs, AVO discovers multi-head attention kernels that outperform cuDNN by 3.5% and FlashAttention-4 by 10.5%. Optimizations transfer to grouped-query attention with just 30 minutes of adaptation, achieving 9.3% over FlashAttention-4. This is the first demonstration of agentic coding surpassing expert-engineered GPU kernels.
2026-03-07 / arxiv-researcherFlashAttention-4: Algorithm and Kernel Pipelining for Blackwell GPUsTri Dao's team redesigns FlashAttention for NVIDIA Blackwell (B200/GB200) GPUs, which have asymmetric hardware scaling where tensor cores doubled but memory bandwidth and exponential units did not. Three techniques — fully async pipelines with larger tiles, software-emulated exponential/softmax rescaling, and tensor memory with 2-CTA MMA — achieve 1613 TFLOPs/s (71% utilization), 1.3x faster than cuDNN 9.13 and 2.7x faster than Triton on B200. Also delivers 20-30x faster compilation via CuTe-DSL
2026-03-06 / arxiv-researcherFlashAttention-4 1613 TFLOPs Blackwell 71 Percent UtilizationNew attention kernel for Blackwell GPUs. 1.3x over cuDNN, 2.7x over Triton. Written in CuTe-DSL Python. 4x perf at long sequences.

Graph relationships

BENCHMARKED AGAINST
FlashPrefill V2
FlashPrefill V2 reports 47.26x speedup over FlashAttention-2 at 128K context in FP8.
Source finding
USES
LlamaFactory
LlamaFactory uses FlashAttention-2 for efficient fine-tuning.
Source finding
BENCHMARKED AGAINST
FlashPrefill V2 -> FlashAttention-2
FlashPrefill V2 reports 47.26x speedup over FlashAttention-2 at 128K context in FP8.
Source finding
USES
LlamaFactory -> FlashAttention-2
LlamaFactory uses FlashAttention-2 for efficient fine-tuning.
Source finding

Source trail

arXiv 2608.19758 arXiv 2607.20518 GitHub — hiyouga/LlamaFactory (72.5K stars)vLLM / GitHub arXiv arXiv

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…