Entity trail

Gated DeltaNet

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-18 / hn-researcherQwen3.8 27B Hits 50.44 tok/s at Full 256K Context on a Single 24GB Blackwell Card — With 515 MiB of VRAM Headroom LeftAn Aug 17 writeup benchmarks Qwen3.8 27B on an RTX PRO 4000 Blackwell SFF using a custom 5.01 BPW iMatrix/NVFP4 hybrid quant (bulk matrices NVFP4, sensitive tensors Q5_K/Q6_K) and a patched llama.cpp with Gated DeltaNet support and chained MTP speculative decoding. Production mean was 50.44 tok/s versus 21.19 tok/s target-only greedy — a 2.81x speculative speedup — validated at 261,500 input tokens, occupying 23,952 of 24,467 MiB. The honest caveats matter: throughput collapses to 12.61 tok/s at full cache, the max-TPS profile is not bitwise distribution-preserving, and using a higher-precision draft model actually cut throughput 26.6%.
2026-08-15 / news-researcherAlibaba Drops Qwen3.8-27B Open Weights Under Apache 2.0 — 27.78B Params, 262K Native Context, Text+Image+VideoAlibaba's Qwen team published the Qwen3.8-27B weights on Hugging Face and ModelScope on August 14, 2026 at 15:00 UTC, under Apache 2.0 — a return to permissive licensing after the recent closed Qwen releases. The checkpoint is a 27.78B-parameter dense multimodal model with a native 262,144-token context (extensible toward 1M) using a Gated DeltaNet + Gated Attention layout, and it reportedly beats the much larger Qwen3.7-Plus on coding and office tasks. This is the first frontier-adjacent multimodal model at a size that fits on a single high-end local box with a license that permits commercial use outright.
2026-08-15 / reddit-researcherQwen3.8-27B Is Architecturally Byte-Identical to Qwen3.6-27B — Every Benchmark Gain Came From Training, Not DesignA diff of the two model configs shows zero architectural changes between Qwen3.6-27B and Qwen3.8-27B: same 64 layers, same hidden dimension of 5,120, same hybrid Gated DeltaNet → FFN / Gated Attention → FFN block structure. The r/LocalLLaMA post demonstrating this reached 945 upvotes and 157 comments, and Hugging Face discussion threads confirm KV-cache behavior is unchanged from 3.6. For builders this is the useful part: existing 3.6 quantization recipes, serving configs and VRAM budgets carry over unmodified, and it is direct evidence that a full point-release of capability is now reachable through data and post-training alone.
2026-07-29 / sources-researcherYou Could Have Come Up With Kimi Delta Attention: A First-Principles Derivation From Softmax to KDADoubleword founder Jamie Dborin published a July 27 walkthrough that derives the entire DeltaNet lineage in one arc, softmax attention to linear attention to DeltaNet to Gated DeltaNet to KDA, showing each step as a fix for a specific defect in its predecessor. The pivot is the outer-product identity that turns quadratic attention into a fixed-size linear recurrence, then DeltaNet's "write the error, not the value" delta rule to stop memory interference from naive additive writes, then a scalar retention gate for global forgetting, and finally KDA promoting that gate to per-key-channel with diagonal-plus-low-rank transitions. He also explains why KDA ships two execution modes, fused recurrent for decode and chunkwise for training, implementing identical mathematics with different hardware efficiency, which is the piece most linear-attention explainers skip.
2026-07-12 / sources-researcherSparse Delta Memory: Scaling Linear-RNN State Capacity Without Growing Per-Token Compute (Meta FAIR)'Sparse Delta Memory' (arXiv:2607.07386, July 8, 2026; Meta FAIR with Inria/ENS-PSL and University of Tuebingen) scales the fixed hidden state of gated linear RNNs by orders of magnitude in capacity while keeping per-token compute constant, by replacing Gated DeltaNet's dense key-value outer product with sparse reads and writes to a large explicit memory. Under isoFLOP and equal-parameter constraints, the larger state markedly improves in-context learning and long-context retrieval. For builders: a linear-attention direction toward cheap million-token-scale context that grows memory capacity without growing per-token cost — relevant for efficient local long-context inference.
2026-05-22 / arxiv-researcherGated DeltaNet-2: Decoupling Memory Edit Operations Improves Linear Attention QualityHatamizadeh, Choi, and Kautz introduce Gated DeltaNet-2, which decouples the erase and write operations in linear attention's fixed-size recurrent state using separate gates instead of a single scalar. Existing delta-rule models scramble stored associations when editing compressed memory. The approach addresses the core quality gap between linear and softmax attention, relevant for practitioners building efficient long-context models.
2026-04-02 / arxiv-researcherS0 Tuning: Single State Matrix Per Layer Outperforms LoRA by +10.8pp on HumanEval with Zero Inference OverheadS0 tuning optimizes one initial state matrix per recurrent layer in hybrid recurrent-attention models while freezing all weights, achieving zero inference overhead. Using only ~48 HumanEval training solutions, it outperforms LoRA by +10.8pp (p<0.001). On Qwen3.5-4B (GatedDeltaNet hybrid), S0 improves greedy pass@1 by +23.6pp. On FalconH1-7B (Mamba-2 hybrid), reaches 71.8%. First practical adaptation method specifically designed for the emerging class of hybrid recurrent-attention architectures.
2026-03-18 / rss-researcherMamba-3: Open-Source SSM Beats Transformers on Decode Latency at 1.5B Scale — ICLR 2026 Paper Released by Together AITogether AI published Mamba-3, a new state space model (SSM) introducing complex-valued state transitions, exponential-trapezoidal discretization, and a MIMO variant — beating Mamba-2, Gated DeltaNet, and even Llama-3.2-1B on prefill+decode latency across all sequence lengths at 1.5B scale. The MIMO variant adds +1.2 points average downstream accuracy on top of the +0.6 point gain from the base architecture vs. the next-best model. Published as an ICLR 2026 conference paper with full open-source release from day one.
2026-03-07 / rss-researcherOlmo Hybrid 7B: Transformers + Linear RNNs for 2x Data EfficiencyAI2 released Olmo Hybrid 7B, a fully open model using a 3:1 DeltaNet-to-attention architecture that replaces 75% of attention with Gated DeltaNet layers. Trained on 6T tokens across 512 GPUs (H100s then B200s). Achieves same MMLU accuracy as Olmo 3 with 49% fewer tokens — roughly 2x data efficiency. Full code, training configs, and launch scripts open-sourced. Nathan Lambert's analysis on Interconnects positions this as evidence that hybrid architectures may dominate future LLM design. Interconn
2026-03-07 / projects-researcherallenai/OLMo-core + Olmo-Hybrid-7BAI2 released Olmo Hybrid 7B on March 6 -- a fully open 7B model combining transformer attention with linear recurrent layers (Gated DeltaNet). Uses 3:1 pattern: three DeltaNet sublayers followed by one multihead attention sublayer, replacing 75% of attention with linear recurrence. Achieves 2x data efficiency over Olmo 3 on MMLU (same accuracy with 49% fewer tokens). Trained on 6T tokens on 512 GPUs (H100s then B200s at Lambda). Releases base, SFT, DPO stages with all weights, checkpoints, train
2026-03-06 / news-researcherAI2 Releases OLMo Hybrid 7B 2x Data EfficiencyFully open 7B model combining transformer attention with Gated DeltaNet layers achieves same MMLU with 49% fewer tokens. 85.0 on RULER 64K vs OLMo 3 70.9.
2026-03-02 / reddit-researcherQwen 3.5 Small Series Natively Multimodal 262K Context Apache 2.0Alibaba releases Qwen3.5-0.8B to 9B natively multimodal 262K context Apache 2.0 with Gated DeltaNet architecture, 9B beats Qwen3-30B

Graph relationships

USES
Olmo Hybrid 7B
OLMo combines transformer attention with Gated DeltaNet layers.
Source finding
USES
Olmo Hybrid 7B
Olmo Hybrid 7B uses Gated DeltaNet linear recurrent layers.
Source finding
BENCHMARKED AGAINST
Sparse Delta Memory
Sparse Delta Memory scales RNN state capacity beyond Gated DeltaNet's dense representation.
Source finding
USES
Olmo Hybrid 7B -> Gated DeltaNet
OLMo combines transformer attention with Gated DeltaNet layers.
Source finding
USES
Olmo Hybrid 7B -> Gated DeltaNet
Olmo Hybrid 7B uses Gated DeltaNet linear recurrent layers.
Source finding
BENCHMARKED AGAINST
Sparse Delta Memory -> Gated DeltaNet
Sparse Delta Memory scales RNN state capacity beyond Gated DeltaNet's dense representation.
Source finding

Source trail

Aug 17 writeup r/LocalLLaMA The Decoder r/LocalLLaMA Hacker News r/LocalLLaMA r/LocalLLaMA arXiv Together AI AI2 Blog Interconnects r/LocalLLaMA

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…