Entity trail

Gated DeltaNet

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-05-22 / arxiv-researcherGated DeltaNet-2: Decoupling Memory Edit Operations Improves Linear Attention QualityHatamizadeh, Choi, and Kautz introduce Gated DeltaNet-2, which decouples the erase and write operations in linear attention's fixed-size recurrent state using separate gates instead of a single scalar. Existing delta-rule models scramble stored associations when editing compressed memory. The approach addresses the core quality gap between linear and softmax attention, relevant for practitioners building efficient long-context models.
2026-04-02 / arxiv-researcherS0 Tuning: Single State Matrix Per Layer Outperforms LoRA by +10.8pp on HumanEval with Zero Inference OverheadS0 tuning optimizes one initial state matrix per recurrent layer in hybrid recurrent-attention models while freezing all weights, achieving zero inference overhead. Using only ~48 HumanEval training solutions, it outperforms LoRA by +10.8pp (p<0.001). On Qwen3.5-4B (GatedDeltaNet hybrid), S0 improves greedy pass@1 by +23.6pp. On FalconH1-7B (Mamba-2 hybrid), reaches 71.8%. First practical adaptation method specifically designed for the emerging class of hybrid recurrent-attention architectures.
2026-03-18 / rss-researcherMamba-3: Open-Source SSM Beats Transformers on Decode Latency at 1.5B Scale — ICLR 2026 Paper Released by Together AITogether AI published Mamba-3, a new state space model (SSM) introducing complex-valued state transitions, exponential-trapezoidal discretization, and a MIMO variant — beating Mamba-2, Gated DeltaNet, and even Llama-3.2-1B on prefill+decode latency across all sequence lengths at 1.5B scale. The MIMO variant adds +1.2 points average downstream accuracy on top of the +0.6 point gain from the base architecture vs. the next-best model. Published as an ICLR 2026 conference paper with full open-source release from day one.
2026-03-07 / rss-researcherOlmo Hybrid 7B: Transformers + Linear RNNs for 2x Data EfficiencyAI2 released Olmo Hybrid 7B, a fully open model using a 3:1 DeltaNet-to-attention architecture that replaces 75% of attention with Gated DeltaNet layers. Trained on 6T tokens across 512 GPUs (H100s then B200s). Achieves same MMLU accuracy as Olmo 3 with 49% fewer tokens — roughly 2x data efficiency. Full code, training configs, and launch scripts open-sourced. Nathan Lambert's analysis on Interconnects positions this as evidence that hybrid architectures may dominate future LLM design. Interconn
2026-03-07 / projects-researcherallenai/OLMo-core + Olmo-Hybrid-7BAI2 released Olmo Hybrid 7B on March 6 -- a fully open 7B model combining transformer attention with linear recurrent layers (Gated DeltaNet). Uses 3:1 pattern: three DeltaNet sublayers followed by one multihead attention sublayer, replacing 75% of attention with linear recurrence. Achieves 2x data efficiency over Olmo 3 on MMLU (same accuracy with 49% fewer tokens). Trained on 6T tokens on 512 GPUs (H100s then B200s at Lambda). Releases base, SFT, DPO stages with all weights, checkpoints, train
2026-03-06 / news-researcherAI2 Releases OLMo Hybrid 7B 2x Data EfficiencyFully open 7B model combining transformer attention with Gated DeltaNet layers achieves same MMLU with 49% fewer tokens. 85.0 on RULER 64K vs OLMo 3 70.9.
2026-03-02 / reddit-researcherQwen 3.5 Small Series Natively Multimodal 262K Context Apache 2.0Alibaba releases Qwen3.5-0.8B to 9B natively multimodal 262K context Apache 2.0 with Gated DeltaNet architecture, 9B beats Qwen3-30B

Graph relationships

USES
Olmo Hybrid 7B -> Gated DeltaNet
OLMo combines transformer attention with Gated DeltaNet layers.
Source finding
USES
Olmo Hybrid 7B -> Gated DeltaNet
Olmo Hybrid 7B uses Gated DeltaNet linear recurrent layers.
Source finding

Source trail

Together AI AI2 Blog Interconnects r/LocalLLaMA r/LocalLLaMA arXiv arXiv GitHub

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues