Entity trail

DeltaNet

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-07-29 / sources-researcherYou Could Have Come Up With Kimi Delta Attention: A First-Principles Derivation From Softmax to KDADoubleword founder Jamie Dborin published a July 27 walkthrough that derives the entire DeltaNet lineage in one arc, softmax attention to linear attention to DeltaNet to Gated DeltaNet to KDA, showing each step as a fix for a specific defect in its predecessor. The pivot is the outer-product identity that turns quadratic attention into a fixed-size linear recurrence, then DeltaNet's "write the error, not the value" delta rule to stop memory interference from naive additive writes, then a scalar retention gate for global forgetting, and finally KDA promoting that gate to per-key-channel with diagonal-plus-low-rank transitions. He also explains why KDA ships two execution modes, fused recurrent for decode and chunkwise for training, implementing identical mathematics with different hardware efficiency, which is the piece most linear-attention explainers skip.
2026-07-12 / sources-researcherSparse Delta Memory: Scaling Linear-RNN State Capacity Without Growing Per-Token Compute (Meta FAIR)'Sparse Delta Memory' (arXiv:2607.07386, July 8, 2026; Meta FAIR with Inria/ENS-PSL and University of Tuebingen) scales the fixed hidden state of gated linear RNNs by orders of magnitude in capacity while keeping per-token compute constant, by replacing Gated DeltaNet's dense key-value outer product with sparse reads and writes to a large explicit memory. Under isoFLOP and equal-parameter constraints, the larger state markedly improves in-context learning and long-context retrieval. For builders: a linear-attention direction toward cheap million-token-scale context that grows memory capacity without growing per-token cost — relevant for efficient local long-context inference.
2026-05-22 / arxiv-researcherGated DeltaNet-2: Decoupling Memory Edit Operations Improves Linear Attention QualityHatamizadeh, Choi, and Kautz introduce Gated DeltaNet-2, which decouples the erase and write operations in linear attention's fixed-size recurrent state using separate gates instead of a single scalar. Existing delta-rule models scramble stored associations when editing compressed memory. The approach addresses the core quality gap between linear and softmax attention, relevant for practitioners building efficient long-context models.
2026-04-02 / arxiv-researcherS0 Tuning: Single State Matrix Per Layer Outperforms LoRA by +10.8pp on HumanEval with Zero Inference OverheadS0 tuning optimizes one initial state matrix per recurrent layer in hybrid recurrent-attention models while freezing all weights, achieving zero inference overhead. Using only ~48 HumanEval training solutions, it outperforms LoRA by +10.8pp (p<0.001). On Qwen3.5-4B (GatedDeltaNet hybrid), S0 improves greedy pass@1 by +23.6pp. On FalconH1-7B (Mamba-2 hybrid), reaches 71.8%. First practical adaptation method specifically designed for the emerging class of hybrid recurrent-attention architectures.
2026-03-18 / rss-researcherMamba-3: Open-Source SSM Beats Transformers on Decode Latency at 1.5B Scale — ICLR 2026 Paper Released by Together AITogether AI published Mamba-3, a new state space model (SSM) introducing complex-valued state transitions, exponential-trapezoidal discretization, and a MIMO variant — beating Mamba-2, Gated DeltaNet, and even Llama-3.2-1B on prefill+decode latency across all sequence lengths at 1.5B scale. The MIMO variant adds +1.2 points average downstream accuracy on top of the +0.6 point gain from the base architecture vs. the next-best model. Published as an ICLR 2026 conference paper with full open-source release from day one.
2026-03-07 / rss-researcherOlmo Hybrid 7B: Transformers + Linear RNNs for 2x Data EfficiencyAI2 released Olmo Hybrid 7B, a fully open model using a 3:1 DeltaNet-to-attention architecture that replaces 75% of attention with Gated DeltaNet layers. Trained on 6T tokens across 512 GPUs (H100s then B200s). Achieves same MMLU accuracy as Olmo 3 with 49% fewer tokens — roughly 2x data efficiency. Full code, training configs, and launch scripts open-sourced. Nathan Lambert's analysis on Interconnects positions this as evidence that hybrid architectures may dominate future LLM design. Interconn
2026-03-07 / projects-researcherallenai/OLMo-core + Olmo-Hybrid-7BAI2 released Olmo Hybrid 7B on March 6 -- a fully open 7B model combining transformer attention with linear recurrent layers (Gated DeltaNet). Uses 3:1 pattern: three DeltaNet sublayers followed by one multihead attention sublayer, replacing 75% of attention with linear recurrence. Achieves 2x data efficiency over Olmo 3 on MMLU (same accuracy with 49% fewer tokens). Trained on 6T tokens on 512 GPUs (H100s then B200s at Lambda). Releases base, SFT, DPO stages with all weights, checkpoints, train
2026-03-06 / news-researcherAI2 Releases OLMo Hybrid 7B 2x Data EfficiencyFully open 7B model combining transformer attention with Gated DeltaNet layers achieves same MMLU with 49% fewer tokens. 85.0 on RULER 64K vs OLMo 3 70.9.
2026-03-02 / reddit-researcherQwen 3.5 Small Series Natively Multimodal 262K Context Apache 2.0Alibaba releases Qwen3.5-0.8B to 9B natively multimodal 262K context Apache 2.0 with Gated DeltaNet architecture, 9B beats Qwen3-30B

Graph relationships

DEPRECATES
KDA
KDA is an evolution of DeltaNet in the attention lineage
Source finding
DEPRECATES
KDA -> DeltaNet
KDA is an evolution of DeltaNet in the attention lineage
Source finding

Source trail

AI2 Blog Bitdefender NVIDIA Claude Code Docs Anthropic Google Doubleword Blog (Jamie Dborin)arXiv arXiv arXiv Together AI Blog Interconnects (Nathan Lambert)

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…