Sparse Delta Memory: Scaling Linear-RNN State Capacity Without Growing Per-Token Compute (Meta FAIR)

arXiv·medium signal

'Sparse Delta Memory' (arXiv:2607.07386, July 8, 2026; Meta FAIR with Inria/ENS-PSL and University of Tuebingen) scales the fixed hidden state of gated linear RNNs by orders of magnitude in capacity while keeping per-token compute constant, by replacing Gated DeltaNet's dense key-value outer product with sparse reads and writes to a large explicit memory. Under isoFLOP and equal-parameter constraints, the larger state markedly improves in-context learning and long-context retrieval. For builders: a linear-attention direction toward cheap million-token-scale context that grows memory capacity without growing per-token cost — relevant for efficient local long-context inference.