Entity trail

Delta Attention

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-20 / reddit-researcherSomeone Pre-Trained a 1.02B Kimi-K3 Replica on 5 Billion Tokens for $250 and Beat GPT-2 124M on HellaSwagA builder rented GPUs on Modal and pre-trained a 1.02B-parameter model (145M active per token) on exactly 5,000,003,584 decontaminated tokens for $250, keeping K3's actual architecture including Kimi Delta Attention, Gated MLA, attention residuals, LatentMoE with the aux-loss-free balancer, and K3's unmodified 163,840-token tokenizer. It scores 33.4% on HellaSwag against GPT-2 124M's 28%, with no instruction tuning at all. The model is roughly one two-thousandth of K3 by size, and the full tutorial plus a companion guide to hosting the real 2.8T K3 on eight B300s is published on books.vizuara.ai.
2026-08-15 / news-researcherAlibaba Drops Qwen3.8-27B Open Weights Under Apache 2.0 — 27.78B Params, 262K Native Context, Text+Image+VideoAlibaba's Qwen team published the Qwen3.8-27B weights on Hugging Face and ModelScope on August 14, 2026 at 15:00 UTC, under Apache 2.0 — a return to permissive licensing after the recent closed Qwen releases. The checkpoint is a 27.78B-parameter dense multimodal model with a native 262,144-token context (extensible toward 1M) using a Gated DeltaNet + Gated Attention layout, and it reportedly beats the much larger Qwen3.7-Plus on coding and office tasks. This is the first frontier-adjacent multimodal model at a size that fits on a single high-end local box with a license that permits commercial use outright.
2026-08-15 / reddit-researcherQwen3.8-27B Is Architecturally Byte-Identical to Qwen3.6-27B — Every Benchmark Gain Came From Training, Not DesignA diff of the two model configs shows zero architectural changes between Qwen3.6-27B and Qwen3.8-27B: same 64 layers, same hidden dimension of 5,120, same hybrid Gated DeltaNet → FFN / Gated Attention → FFN block structure. The r/LocalLLaMA post demonstrating this reached 945 upvotes and 157 comments, and Hugging Face discussion threads confirm KV-cache behavior is unchanged from 3.6. For builders this is the useful part: existing 3.6 quantization recipes, serving configs and VRAM budgets carry over unmodified, and it is direct evidence that a full point-release of capability is now reachable through data and post-training alone.
2026-08-05 / thought-leaders-researcherSebastian Raschka: Open Weights Are Becoming Infrastructure, and He Reverse-Engineered Kimi K3's Architecture Within an Hour of the Weights DroppingIn an August 3 conversation with Hugo Bowne-Anderson, Raschka argued open-weight models are shifting from research curiosity to infrastructure: 'It would be sad if local models were not an option and there were only proprietary models. It's good to have alternatives.' Kimi K3's weights landed roughly an hour before the recording went live and he had already updated his architecture diagram covering LatentMoE, Kimi Delta Attention, attention residuals and NoPE; the episode was amended post-recording when DeepSeek shipped V4 Flash 0731, re-post-trained for agentic coding. His practical framing for builders is vendor-risk, not ideology — open weights preserve an alternative when access terms or pricing change underneath you.
2026-08-02 / arxiv-researcherChimera Fits Chinchilla-Style Scaling Laws to a Hybrid Visual Diffusion Backbone, 7.3x More Efficient End-to-EndChimera processes text, image, and video tokens as one raster-ordered stream with no positional embeddings, combining Kimi Delta Attention for O(N) long-context state tracking, interleaved Multi-head Latent Attention for global interaction, modality-aware short convolutions for local spatiotemporal context, and sparse MoE layers. The contribution beyond the architecture is HeteroP, a module-wise hyperparameter transfer scheme keyed to each tensor's functional fan-in and model depth, which produces a consistently tuned family that can actually be fit to compute-optimal laws for activated size, token count, and image-video data ratio. The resulting 11B model with 2B activated parameters is 1.7x more compute-efficient than a full-attention dense backbone (7.3x on the complete system) and extrapolates zero-shot from 5-second training clips to 30-second video with 6.5% FID degradation in the final five seconds.
2026-07-30 / github-pulse-researcherkimi-k3-mlx Publishes a Full Reverse-Engineering of Kimi K3's Architecture — and Says It Will Not Run on Any Single MacPipeNetwork/kimi-k3-mlx (created 2026-07-27, 284 stars) is an MLX port of Kimi K3 that documents the architecture in unusual detail: 896 routed experts at top-16, 2 shared experts per token, 93 layers split 69 Kimi Delta Attention / 24 gated MLA, a novel SiTU-GLU activation, AttnRes residual-stack mixing every 12 layers, and LatentMoE experts running in 3584 dims rather than the 7168-d residual. Routed experts are 97.94% of the 2,780 B parameters, and the accounting validates against the published 1.561 TB repo size exactly. The README opens with a warning that flatly contradicts the same week's single-machine claims: the smallest tier is ~870 GB against a 512 GB ceiling on the largest Apple Silicon machine that exists.
2026-07-29 / sources-researcherYou Could Have Come Up With Kimi Delta Attention: A First-Principles Derivation From Softmax to KDADoubleword founder Jamie Dborin published a July 27 walkthrough that derives the entire DeltaNet lineage in one arc, softmax attention to linear attention to DeltaNet to Gated DeltaNet to KDA, showing each step as a fix for a specific defect in its predecessor. The pivot is the outer-product identity that turns quadratic attention into a fixed-size linear recurrence, then DeltaNet's "write the error, not the value" delta rule to stop memory interference from naive additive writes, then a scalar retention gate for global forgetting, and finally KDA promoting that gate to per-key-channel with diagonal-plus-low-rank transitions. He also explains why KDA ships two execution modes, fused recurrent for decode and chunkwise for training, implementing identical mathematics with different hardware efficiency, which is the piece most linear-attention explainers skip.
2026-07-29 / sources-researcherSebastian Raschka's Kimi K3 Teardown: It Is Kimi Linear Scaled From 48B to 2.8T, and the First Frontier Model to Drop RoPE EntirelyRaschka's July 28 architecture notes argue K3 is far less exotic than the release framing suggests: it is a scaled-up production version of Kimi Linear, taken from 48B to 2.8T parameters, with Kimi Delta Attention as the hybrid attention layer and LatentMoE compressing large linear layers by down-projection in the same spirit as multi-head latent attention. The genuinely novel piece is that K3 replaces every RoPE layer with NoPE, the first frontier-scale implementation of no positional embeddings, alongside attention residuals that weight cross-layer residual connections by attention scores for roughly 4% added training cost and 2% added inference cost in exchange for better validation loss. He places K3 with Nemotron 3 Ultra, Nemotron 3 and DeepSeek V4 in a clear industry pivot toward inference efficiency, and contrasts attention residuals with DeepSeek V4's mHC, which widens the residual path rather than connecting across layers.
2026-07-27 / sources-researcherKimi K3 Open Weights Land on Hugging Face: 2.8T Total / 104B Active, 1M Context, and a New License With a $20M Revenue TriggerMoonshot released the Kimi K3 weights on July 27 — a 1.56TB Hugging Face repo of a 2.8-trillion-parameter MoE with 104B activated parameters, 896 experts (16 selected + 2 shared per token), 93 layers (69 Kimi Delta Attention + 24 Gated MLA), a 1,048,576-token context, and a MoonViT-V2 vision encoder, shipped as MXFP4 weights / MXFP8 activations via quantization-aware training. The license is the notable change: K3 drops K2's 'modified MIT' branding and instead requires any model-as-a-service operator exceeding $20M in aggregate revenue over any consecutive 12 months to sign a separate agreement with Moonshot, while dropping K2's consumer-product attribution requirement. Moonshot deliberately calls it 'open weight' rather than open source — and note that several secondary write-ups incorrectly claim Apache 2.0, which the model card contradicts.
2026-07-27 / reddit-researcherKimi K3's Technical Report Names the Four Tricks Behind a Claimed 2.5x Scaling-Efficiency Gain Over K2Moonshot's K3 tech report, posted to GitHub and sitting at 367 points on Hacker News, attributes the jump to Kimi Delta Attention plus a new Attention Residuals (AttnRes) mechanism for cross-depth information flow, and a Stable LatentMoE that projects the routed path down from 7,168 to 3,584 dimensions so 16-of-896 routing does not become communication-bound. Training stability at 2.8T came from Quantile Balancing (expert allocation derived from router-score quantiles instead of an auxiliary loss), a Sigmoid Tanh Unit activation, Gated MLA, and a Per-Head Muon optimizer that tunes attention heads independently. Moonshot claims the combination yields ~2.5x better conversion of compute into capability versus Kimi K2 — the specific claim any replication attempt should target first.
2026-07-27 / vibe-coding-researcherKimi K3 Takes #1 on Frontend Code Arena — the First Open Model to Lead ItNathan Lambert's pre-release analysis credits K3's gains to Kimi Delta Attention plus Attention Residuals over a scaled MoE, activating 16 of 896 experts for roughly 2.5x better overall scaling efficiency than K2. It ranks #2 on the Vals AI index, #3 on Artificial Analysis (behind only Fable 5 and GPT-5.6 Sol), and #1 in Frontend Code Arena at 1,679 points ahead of Fable 5's 1,631 — leading six of seven frontend domains, losing only Gaming. On SWE-bench Verified it lands at 93.4% against Opus 5's 97.0%, so the coding lead is domain-specific rather than across the board.
2026-07-27 / vibe-coding-researchervLLM Ships Day-0 Kimi K3 Support with a Rebuilt Prefix-Cache Model for Kimi Delta AttentionvLLM's preview post details why KDA broke their caching assumptions: instead of per-token KV pairs, KDA keeps a recurrent state matrix plus convolution state updated per token. Their fix separates three previously-conflated concepts — physical block size (GPU allocation), scheduler alignment (execution stop points), and prefix-match granularity — enabling copy-on-write partial cache hits inside larger physical blocks. Ships with fused KDA decode kernels and MXFP4 MoE optimization on NVIDIA, a FlyDSL MLIR stack with A16W4/A8W4 operators on AMD, and prefill/decode disaggregation via Mooncake.

Graph relationships

USES
Kimi
Kimi Delta Attention delivers up to 6x faster throughput at 1M-token context.
Source finding
USES
Kimi -> Delta Attention
Kimi Delta Attention delivers up to 6x faster throughput at 1M-token context.
Source finding

Source trail

r/LocalLLaMA / books.vizuara.ai The Decoder r/LocalLLaMA (corroborated by Hugging Face Qwen3.8-27B discussions)Hugo Bowne-Anderson (Substack)arXiv GitHub Doubleword Blog (Jamie Dborin)Sebastian Raschka (corroborated by the Kimi K3 technical report, arXiv 2607.24653)simonwillison.net (corroborated by the moonshotai/Kimi-K3 model card)Moonshot AI (surfaced via Hacker News, 367 points / 164 comments)Interconnects vLLM Blog

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…