Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-31 · high
The paper says matching those gains with linear attention would require training from scratch or heavy post-training.
Why now: The comparison arrives as linear-attention retrofits keep getting pitched as a memory-saving swap for long-context models.
Sliding window attention with sinks beats post-trained linear attention on long-context retrieval. It scores 2 to 10 times higher on the Needle-in-a-Haystack and BABILong benchmarks, per a new comparison of attention mechanisms that tested both approaches across multiple LLMs and downstream tasks.
That gap matters for anyone weighing how to cut inference memory costs on long-context models. Linear attention has become a popular retrofit for that exact problem: take an existing model, post-train it into a linear-attention variant, and save memory without retraining from scratch.
The comparison ran the baseline retrofit papers tend to skip. It tested SWA with sinks directly against post-trained linear models on the same tasks, not just the two retrieval benchmarks but the broader set of downstream tasks too. SWA matched or beat linear attention across the board.
The paper's recommendation is blunt. Switch to SWA for the memory savings. Matching its retrieval scores with a linear model would take training from scratch or heavy post-training, the paper says, not a light retrofit.
It's a negative result for a popular idea. Papers proposing linear-attention retrofits don't tend to run this comparison, which is exactly why it's useful now.
Each link below shares sources, entities, or timing with this story.
Needle benchmarked against Gemini / Shared entity: LLMs / Same source domain / Earlier coverage
Linked by a graph relationship (Needle benchmarked against Gemini); both cover LLMs; reported by the same outlet (arxiv.org).
Linked by a graph relationship (Needle benchmarked against Gemini); both cover LLMs; reported by the same outlet (arxiv.org).
Shared entity: LLMs / Same source domain / Shared topic / Earlier coverage
Both cover LLMs; reported by the same outlet (arxiv.org); overlapping topics (baseline, beat).
Both cover LLMs; reported by the same outlet (arxiv.org); overlapping topics (beat, higher).
Both cover LLMs; reported by the same outlet (arxiv.org); overlapping topics (attention, kind).
Shared entity: LLMs / Same source domain / Earlier coverage / Tension
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-08-17.
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-08-07.
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-07-31.