Gated DeltaNet-2: Decoupling Memory Edit Operations Improves Linear Attention Quality

arXiv·medium signal

Hatamizadeh, Choi, and Kautz introduce Gated DeltaNet-2, which decouples the erase and write operations in linear attention's fixed-size recurrent state using separate gates instead of a single scalar. Existing delta-rule models scramble stored associations when editing compressed memory. The approach addresses the core quality gap between linear and softmax attention, relevant for practitioners building efficient long-context models.