Meta FAIR's Sparse Delta Memory grows RNN state capacity without growing per-token cost.

It scales the fixed hidden state of gated linear RNNs by orders of magnitude by replacing Gated DeltaNet's dense key-value outer product with sparse reads and writes to a large explicit memory (arXiv). Under isoFLOP and equal-parameter constraints, the bigger state markedly improves in-context learning and long-context retrieval. This is a real direction toward cheap million-token context on local hardware, where memory capacity scales but per-token compute stays flat.