Sliding Window Attention With Sinks Beats Post-Trained Linear Attention by 2x to 10x on Long-Context Retrieval, With No Post-Training

arXiv 2608.28444·high signalXBlueskyLinkedInCopy link

arXiv 2608.28444 (2026-08-28, cs.CL/cs.LG) compares retrofitted Linear Attention against the simpler baseline the literature skipped. Across multiple LLMs and downstream tasks, Sliding Window Attention with sinks matches or beats post-trained Linear Attention models, and on long-context reasoning benchmarks Needle-in-a-Haystack and BABILong it scores 2 to 10 times higher. The practical recommendation is blunt: SWA needs no post-training, is fast and low-memory, so switch to it for inference memory cost rather than post-training a linear model, which the authors argue would need training from scratch or extensive post-training just to match.