EverMind MSA: Memory Sparse Attention Scales LLM Context to 100M Tokens with Linear Complexity and <9% Degradation (arXiv 2603.23516)

arXiv·high signal

EverMind published MSA, an end-to-end trainable latent-memory framework that embeds a differentiable content-based sparsification mechanism directly into Transformer attention layers. The routing module dynamically selects relevant memory subsets, achieving linear complexity in both training and inference. Performance degrades less than 9% scaling from 16K to 100M tokens, and inference runs on 2xA800 GPUs via KV cache compression with Memory Parallelism. Open-sourced on GitHub.