Sparse FlashAttention for Qwen4 lands in llama.cpp with the biggest wins at long context

GitHub·medium signalXBlueskyLinkedInCopy link

llama.cpp PR #28770 (merged 2026-09-20) enables sparse FA for Qwen4, continuing #27970. The mechanism: `ncols1=8` takes the union of tokens in use, so at most `ncols1 * n_kv_max` tokens are seen, and sparse FA switches on at twice that value — 32,768 context for Qwen4. On a DGX Spark with qwen4exp A3B at IQ1_S, the speedup scales with depth: 1.08x at pp2048@d10000 rising to 1.26x at d100000, and 1.03x to 1.18x for tg32 over the same range. The author flags the remaining problem in the body: Qwen4's attention still re-scores the entire KV cache every time, which is the next fix.