Entity trail

Attention Residuals

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-05 / thought-leaders-researcherSebastian Raschka: Open Weights Are Becoming Infrastructure, and He Reverse-Engineered Kimi K3's Architecture Within an Hour of the Weights DroppingIn an August 3 conversation with Hugo Bowne-Anderson, Raschka argued open-weight models are shifting from research curiosity to infrastructure: 'It would be sad if local models were not an option and there were only proprietary models. It's good to have alternatives.' Kimi K3's weights landed roughly an hour before the recording went live and he had already updated his architecture diagram covering LatentMoE, Kimi Delta Attention, attention residuals and NoPE; the episode was amended post-recording when DeepSeek shipped V4 Flash 0731, re-post-trained for agentic coding. His practical framing for builders is vendor-risk, not ideology — open weights preserve an alternative when access terms or pricing change underneath you.
2026-07-29 / sources-researcherSebastian Raschka's Kimi K3 Teardown: It Is Kimi Linear Scaled From 48B to 2.8T, and the First Frontier Model to Drop RoPE EntirelyRaschka's July 28 architecture notes argue K3 is far less exotic than the release framing suggests: it is a scaled-up production version of Kimi Linear, taken from 48B to 2.8T parameters, with Kimi Delta Attention as the hybrid attention layer and LatentMoE compressing large linear layers by down-projection in the same spirit as multi-head latent attention. The genuinely novel piece is that K3 replaces every RoPE layer with NoPE, the first frontier-scale implementation of no positional embeddings, alongside attention residuals that weight cross-layer residual connections by attention scores for roughly 4% added training cost and 2% added inference cost in exchange for better validation loss. He places K3 with Nemotron 3 Ultra, Nemotron 3 and DeepSeek V4 in a clear industry pivot toward inference efficiency, and contrasts attention residuals with DeepSeek V4's mHC, which widens the residual path rather than connecting across layers.
2026-07-27 / reddit-researcherKimi K3's Technical Report Names the Four Tricks Behind a Claimed 2.5x Scaling-Efficiency Gain Over K2Moonshot's K3 tech report, posted to GitHub and sitting at 367 points on Hacker News, attributes the jump to Kimi Delta Attention plus a new Attention Residuals (AttnRes) mechanism for cross-depth information flow, and a Stable LatentMoE that projects the routed path down from 7,168 to 3,584 dimensions so 16-of-896 routing does not become communication-bound. Training stability at 2.8T came from Quantile Balancing (expert allocation derived from router-score quantiles instead of an auxiliary loss), a Sigmoid Tanh Unit activation, Gated MLA, and a Per-Head Muon optimizer that tunes attention heads independently. Moonshot claims the combination yields ~2.5x better conversion of compute into capability versus Kimi K2 — the specific claim any replication attempt should target first.
2026-07-27 / vibe-coding-researcherKimi K3 Takes #1 on Frontend Code Arena — the First Open Model to Lead ItNathan Lambert's pre-release analysis credits K3's gains to Kimi Delta Attention plus Attention Residuals over a scaled MoE, activating 16 of 896 experts for roughly 2.5x better overall scaling efficiency than K2. It ranks #2 on the Vals AI index, #3 on Artificial Analysis (behind only Fable 5 and GPT-5.6 Sol), and #1 in Frontend Code Arena at 1,679 points ahead of Fable 5's 1,631 — leading six of seven frontend domains, losing only Gaming. On SWE-bench Verified it lands at 93.4% against Opus 5's 97.0%, so the coding lead is domain-specific rather than across the board.
2026-07-25 / sources-researcherSANA-Video 2.0 Generates 720p Video on a Single GPU in ~13 Seconds Using a 3:1 Linear-to-Softmax Attention RatioSANA-Video 2.0 (arXiv 2607.21553, 23 July, Junsong Chen plus 13 co-authors) uses gated linear attention for O(N)-dominated mixing with periodic gated-softmax anchors at a 3:1 ratio, plus Block Attention Residuals that raise effective rank about 12%. The 5B and 14B models produce 720p on a single GPU: 13.06 seconds for a 720p 5-second clip fully optimized, 13.2 seconds for 40-step sampling on an H100, with a compiled DiT forward pass 3.2× faster than the full-softmax baseline at 720p and a VBench score of 84.30 at 480p. The authors claim roughly 120× the throughput of Wan 2.2-A14B — the hybrid-attention recipe is the transferable idea here, not just the video results.
2026-07-21 / thought-leaders-researcherMoonshot Ships Kimi K3 — 2.8 Trillion Parameters, First Open 3T-Class Model, and It Beats Fable 5 on Arena's Frontend Code BoardMoonshot AI released Kimi K3, a sparse MoE activating just 16 of 896 experts per token (~1.8% of the pool) with a 1M-token context window and native vision, introducing two new architectural pieces — Kimi Delta Attention and Attention Residuals. It ranked first in Arena's Frontend Code evaluation at 1,679 points, ahead of Claude Fable 5 in blind developer testing, while trailing Fable 5 and GPT-5.6 Sol on overall capability. Full weights land by July 27 under a Modified-MIT-style license at $0.30/M cache-hit input, $3/M cache-miss, $15/M output — meaning a frontier-adjacent coding model becomes self-hostable in under a week.
2026-03-21 / reddit-researcherKimi Publishes 'Attention Residuals' Paper — Proposes Replacing Standard Transformer Residual ConnectionsMoonshot AI (Kimi) published a paper proposing 'attention residuals' to replace the residual connections used in every transformer since the ResNet era — routing skip-connection information through learned attention weights instead of direct addition. The r/LocalLLaMA post (117 upvotes) reports results 'look legit' at the scales tested. If the architecture validates at frontier scale, this would be a meaningful departure from core transformer topology that has been stable since 2017.
2026-03-17 / sources-researcherAttention Residuals (Moonshot AI / Kimi): Drop-In Residual Replacement Shows Consistent Scaling Improvements — Already in Production at 48B ScaleMoonshot AI's Kimi team proposes replacing standard fixed residual connections in transformers with softmax attention over preceding layer outputs — letting each layer selectively weight earlier representations rather than blindly accumulating them. A practical 'Block AttnRes' variant partitions layers into blocks for memory efficiency and validates consistent improvement across model sizes via scaling laws. Kimi Linear (48B total / 3B active, trained on 1.4T tokens) already uses this in production; the paper is the #1 trending on HuggingFace with 1,330 upvotes.
2026-03-17 / vibe-coding-researcherKimi Attention Residuals: Learned Per-Layer Weights Replace Fixed Unit Accumulation in Transformer ResidualsThe Kimi team published arXiv:2603.15031 proposing learned per-layer scalar weights for residual connections, replacing the standard fixed weight-1.0 accumulation in PreNorm transformers. Standard LLMs sum all layer outputs equally; learned weights allow the model to selectively weight each layer's contribution. Potential drop-in architectural improvement for any transformer stack.
2026-03-16 / reddit-researcherMoonshot AI Introduces Attention Residuals (AttnRes) — Replaces Fixed Residual Connections with Learned Depth-Wise AttentionKimi's team announced Attention Residuals on March 16: instead of uniformly adding each layer's output to a shared hidden state, AttnRes computes weighted combinations over all previous layers — treating model depth like a sequence dimension. This lets each layer selectively retrieve useful representations from earlier points, addressing the 'equal blending' limitation of standard residuals. Targets improved scaling stability, information preservation in deep networks, and context retrieval in large-scale MoE architectures.

Graph relationships

RELEASED
Moonshot AI
Kimi's team announced Attention Residuals on March 16 as an enhancement to residual connections.
Source finding
RELEASED
Moonshot AI
Moonshot AI published the Attention Residuals paper.
Source finding
RELEASED
Moonshot AI -> Attention Residuals
Kimi's team announced Attention Residuals on March 16 as an enhancement to residual connections.
Source finding
RELEASED
Moonshot AI -> Attention Residuals
Moonshot AI published the Attention Residuals paper.
Source finding

Source trail

Nathan Lambert's analysis Kimi K3 The Verge framed it told Fox Business on July 21 MIT Technology Review reports lost its director after three months voluntary frontier-model framework attention residuals Source Hugo Bowne-Anderson (Substack)Sebastian Raschka (corroborated by the Kimi K3 technical report, arXiv 2607.24653)Moonshot AI (surfaced via Hacker News, 367 points / 164 comments)

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…