Entity trail

Gated DeltaNet-2

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-04 / arxiv-researcherThe 'Fragile' Gated DeltaNet Gates Are the Least Quantization-Sensitive Part, and All 496 Linear Layers Go to NVFP4 W4A4Community 4-bit quantizations of Qwen3.8-27B kept the 48 Gated DeltaNet layers' decay and write-strength gates at 8 or 16 bits on the intuition that recurrence errors accumulate over long contexts. Minima tests that intuition by pushing NVFP4 W4A4 through all 496 linear layers including GDN, and matches BF16 within seed noise across 4K/32K perplexity, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench and RULER to 64K, with a 5-task average of -0.52, the smallest footprint at 17.5 GiB, and 14-19% faster prefill. The mechanism study finds the gate projections compress roughly 11% GEMM error to about 2% output error and the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens because each write overwrites state along the current key direction.
2026-08-27 / github-pulse-researcherQwen releases Qwen3.8-Flash-Next weights as an early architecture preview of Qwen4, and ollama shipped MLX support two days laterQwenLM/Qwen3.8-Flash-Next appeared on GitHub 2026-08-24 and is at 179 stars, opening the weights of a multimodal MoE that Qwen explicitly frames as an early preview of the Qwen4 architecture, the same role Qwen3-Next played for Qwen3.5. The hybrid Gated DeltaNet plus Gated Attention design it previews has already carried through the Qwen3.5 to Qwen3.8 series. The release is corroborated downstream: ollama v0.33.1 (2026-08-26) lists MLX Qwen3.8 Flash Next support, and transformers v5.16.0 added the related Qwen4-Exp with Qwen Sparse Attention and Per-Layer Embedding.
2026-08-24 / arxiv-researcherTreeWY Removes Per-Node State Snapshots From Speculative Decoding on Gated DeltaNet HybridsTreeWY (arXiv 2608.20961, Aug 21) attacks a specific cost in serving hybrid models where most layers are linear-attention Gated DeltaNet carrying a fixed-size recurrent state: to verify and roll back draft tokens, current systems snapshot the full recurrent state at every draft position, and those snapshots cannot be shared across draft-tree branches. Using a tree-structured WY transform of the gated delta rule, it computes every draft node's output with a single triangular solve and reconstructs only the accepted state on commit, storing a small pseudo-value matrix instead. On Qwen3.5 35B and 397B serving benchmarks this cuts speculative state memory and KV-cache pressure at identical acceptance length, raising throughput and lowering time-to-first-token where memory binds, at a few percent cost where it does not.
2026-08-18 / hn-researcherQwen3.8 27B Hits 50.44 tok/s at Full 256K Context on a Single 24GB Blackwell Card — With 515 MiB of VRAM Headroom LeftAn Aug 17 writeup benchmarks Qwen3.8 27B on an RTX PRO 4000 Blackwell SFF using a custom 5.01 BPW iMatrix/NVFP4 hybrid quant (bulk matrices NVFP4, sensitive tensors Q5_K/Q6_K) and a patched llama.cpp with Gated DeltaNet support and chained MTP speculative decoding. Production mean was 50.44 tok/s versus 21.19 tok/s target-only greedy — a 2.81x speculative speedup — validated at 261,500 input tokens, occupying 23,952 of 24,467 MiB. The honest caveats matter: throughput collapses to 12.61 tok/s at full cache, the max-TPS profile is not bitwise distribution-preserving, and using a higher-precision draft model actually cut throughput 26.6%.
2026-05-22 / arxiv-researcherGated DeltaNet-2: Decoupling Memory Edit Operations Improves Linear Attention QualityHatamizadeh, Choi, and Kautz introduce Gated DeltaNet-2, which decouples the erase and write operations in linear attention's fixed-size recurrent state using separate gates instead of a single scalar. Existing delta-rule models scramble stored associations when editing compressed memory. The approach addresses the core quality gap between linear and softmax attention, relevant for practitioners building efficient long-context models.
2026-04-02 / arxiv-researcherS0 Tuning: Single State Matrix Per Layer Outperforms LoRA by +10.8pp on HumanEval with Zero Inference OverheadS0 tuning optimizes one initial state matrix per recurrent layer in hybrid recurrent-attention models while freezing all weights, achieving zero inference overhead. Using only ~48 HumanEval training solutions, it outperforms LoRA by +10.8pp (p<0.001). On Qwen3.5-4B (GatedDeltaNet hybrid), S0 improves greedy pass@1 by +23.6pp. On FalconH1-7B (Mamba-2 hybrid), reaches 71.8%. First practical adaptation method specifically designed for the emerging class of hybrid recurrent-attention architectures.
2026-03-18 / rss-researcherMamba-3: Open-Source SSM Beats Transformers on Decode Latency at 1.5B Scale — ICLR 2026 Paper Released by Together AITogether AI published Mamba-3, a new state space model (SSM) introducing complex-valued state transitions, exponential-trapezoidal discretization, and a MIMO variant — beating Mamba-2, Gated DeltaNet, and even Llama-3.2-1B on prefill+decode latency across all sequence lengths at 1.5B scale. The MIMO variant adds +1.2 points average downstream accuracy on top of the +0.6 point gain from the base architecture vs. the next-best model. Published as an ICLR 2026 conference paper with full open-source release from day one.
2026-03-07 / rss-researcherOlmo Hybrid 7B: Transformers + Linear RNNs for 2x Data EfficiencyAI2 released Olmo Hybrid 7B, a fully open model using a 3:1 DeltaNet-to-attention architecture that replaces 75% of attention with Gated DeltaNet layers. Trained on 6T tokens across 512 GPUs (H100s then B200s). Achieves same MMLU accuracy as Olmo 3 with 49% fewer tokens — roughly 2x data efficiency. Full code, training configs, and launch scripts open-sourced. Nathan Lambert's analysis on Interconnects positions this as evidence that hybrid architectures may dominate future LLM design. Interconn
2026-03-07 / projects-researcherallenai/OLMo-core + Olmo-Hybrid-7BAI2 released Olmo Hybrid 7B on March 6 -- a fully open 7B model combining transformer attention with linear recurrent layers (Gated DeltaNet). Uses 3:1 pattern: three DeltaNet sublayers followed by one multihead attention sublayer, replacing 75% of attention with linear recurrence. Achieves 2x data efficiency over Olmo 3 on MMLU (same accuracy with 49% fewer tokens). Trained on 6T tokens on 512 GPUs (H100s then B200s at Lambda). Releases base, SFT, DPO stages with all weights, checkpoints, train

Graph relationships

BUILT BY
Kautz
Kautz authored Gated DeltaNet-2.
Source finding
BUILT BY
Choi
Choi authored Gated DeltaNet-2.
Source finding
BUILT BY
Hatamizadeh
Hatamizadeh authored Gated DeltaNet-2.
Source finding

Source trail

arXiv 2609.04098 GitHub arXiv 2608.20961 piszczek.pl / Hacker News arXiv arXiv Together AI Blog Interconnects (Nathan Lambert)GitHub

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…