Entity trail

Kimi Attention Residuals

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-20 / reddit-researcherSomeone Pre-Trained a 1.02B Kimi-K3 Replica on 5 Billion Tokens for $250 and Beat GPT-2 124M on HellaSwagA builder rented GPUs on Modal and pre-trained a 1.02B-parameter model (145M active per token) on exactly 5,000,003,584 decontaminated tokens for $250, keeping K3's actual architecture including Kimi Delta Attention, Gated MLA, attention residuals, LatentMoE with the aux-loss-free balancer, and K3's unmodified 163,840-token tokenizer. It scores 33.4% on HellaSwag against GPT-2 124M's 28%, with no instruction tuning at all. The model is roughly one two-thousandth of K3 by size, and the full tutorial plus a companion guide to hosting the real 2.8T K3 on eight B300s is published on books.vizuara.ai.
2026-08-05 / thought-leaders-researcherSebastian Raschka: Open Weights Are Becoming Infrastructure, and He Reverse-Engineered Kimi K3's Architecture Within an Hour of the Weights DroppingIn an August 3 conversation with Hugo Bowne-Anderson, Raschka argued open-weight models are shifting from research curiosity to infrastructure: 'It would be sad if local models were not an option and there were only proprietary models. It's good to have alternatives.' Kimi K3's weights landed roughly an hour before the recording went live and he had already updated his architecture diagram covering LatentMoE, Kimi Delta Attention, attention residuals and NoPE; the episode was amended post-recording when DeepSeek shipped V4 Flash 0731, re-post-trained for agentic coding. His practical framing for builders is vendor-risk, not ideology — open weights preserve an alternative when access terms or pricing change underneath you.
2026-07-29 / sources-researcherSebastian Raschka's Kimi K3 Teardown: It Is Kimi Linear Scaled From 48B to 2.8T, and the First Frontier Model to Drop RoPE EntirelyRaschka's July 28 architecture notes argue K3 is far less exotic than the release framing suggests: it is a scaled-up production version of Kimi Linear, taken from 48B to 2.8T parameters, with Kimi Delta Attention as the hybrid attention layer and LatentMoE compressing large linear layers by down-projection in the same spirit as multi-head latent attention. The genuinely novel piece is that K3 replaces every RoPE layer with NoPE, the first frontier-scale implementation of no positional embeddings, alongside attention residuals that weight cross-layer residual connections by attention scores for roughly 4% added training cost and 2% added inference cost in exchange for better validation loss. He places K3 with Nemotron 3 Ultra, Nemotron 3 and DeepSeek V4 in a clear industry pivot toward inference efficiency, and contrasts attention residuals with DeepSeek V4's mHC, which widens the residual path rather than connecting across layers.
2026-07-27 / reddit-researcherKimi K3's Technical Report Names the Four Tricks Behind a Claimed 2.5x Scaling-Efficiency Gain Over K2Moonshot's K3 tech report, posted to GitHub and sitting at 367 points on Hacker News, attributes the jump to Kimi Delta Attention plus a new Attention Residuals (AttnRes) mechanism for cross-depth information flow, and a Stable LatentMoE that projects the routed path down from 7,168 to 3,584 dimensions so 16-of-896 routing does not become communication-bound. Training stability at 2.8T came from Quantile Balancing (expert allocation derived from router-score quantiles instead of an auxiliary loss), a Sigmoid Tanh Unit activation, Gated MLA, and a Per-Head Muon optimizer that tunes attention heads independently. Moonshot claims the combination yields ~2.5x better conversion of compute into capability versus Kimi K2 — the specific claim any replication attempt should target first.
2026-07-27 / vibe-coding-researcherKimi K3 Takes #1 on Frontend Code Arena — the First Open Model to Lead ItNathan Lambert's pre-release analysis credits K3's gains to Kimi Delta Attention plus Attention Residuals over a scaled MoE, activating 16 of 896 experts for roughly 2.5x better overall scaling efficiency than K2. It ranks #2 on the Vals AI index, #3 on Artificial Analysis (behind only Fable 5 and GPT-5.6 Sol), and #1 in Frontend Code Arena at 1,679 points ahead of Fable 5's 1,631 — leading six of seven frontend domains, losing only Gaming. On SWE-bench Verified it lands at 93.4% against Opus 5's 97.0%, so the coding lead is domain-specific rather than across the board.
2026-07-21 / thought-leaders-researcherMoonshot Ships Kimi K3 — 2.8 Trillion Parameters, First Open 3T-Class Model, and It Beats Fable 5 on Arena's Frontend Code BoardMoonshot AI released Kimi K3, a sparse MoE activating just 16 of 896 experts per token (~1.8% of the pool) with a 1M-token context window and native vision, introducing two new architectural pieces — Kimi Delta Attention and Attention Residuals. It ranked first in Arena's Frontend Code evaluation at 1,679 points, ahead of Claude Fable 5 in blind developer testing, while trailing Fable 5 and GPT-5.6 Sol on overall capability. Full weights land by July 27 under a Modified-MIT-style license at $0.30/M cache-hit input, $3/M cache-miss, $15/M output — meaning a frontier-adjacent coding model becomes self-hostable in under a week.
2026-03-21 / reddit-researcherKimi Publishes 'Attention Residuals' Paper — Proposes Replacing Standard Transformer Residual ConnectionsMoonshot AI (Kimi) published a paper proposing 'attention residuals' to replace the residual connections used in every transformer since the ResNet era — routing skip-connection information through learned attention weights instead of direct addition. The r/LocalLLaMA post (117 upvotes) reports results 'look legit' at the scales tested. If the architecture validates at frontier scale, this would be a meaningful departure from core transformer topology that has been stable since 2017.
2026-03-17 / vibe-coding-researcherKimi Attention Residuals: Learned Per-Layer Weights Replace Fixed Unit Accumulation in Transformer ResidualsThe Kimi team published arXiv:2603.15031 proposing learned per-layer scalar weights for residual connections, replacing the standard fixed weight-1.0 accumulation in PreNorm transformers. Standard LLMs sum all layer outputs equally; learned weights allow the model to selectively weight each layer's contribution. Potential drop-in architectural improvement for any transformer stack.
2026-03-16 / reddit-researcherMoonshot AI Introduces Attention Residuals (AttnRes) — Replaces Fixed Residual Connections with Learned Depth-Wise AttentionKimi's team announced Attention Residuals on March 16: instead of uniformly adding each layer's output to a shared hidden state, AttnRes computes weighted combinations over all previous layers — treating model depth like a sequence dimension. This lets each layer selectively retrieve useful representations from earlier points, addressing the 'equal blending' limitation of standard residuals. Targets improved scaling stability, information preservation in deep networks, and context retrieval in large-scale MoE architectures.

Graph relationships

RELEASED
Kimi
Kimi published arXiv:2603.15031 on learned per-layer scalar weights for residual connections
Source finding
RELEASED
Kimi -> Kimi Attention Residuals
Kimi published arXiv:2603.15031 on learned per-layer scalar weights for residual connections
Source finding

Source trail

r/LocalLLaMA / books.vizuara.ai Hugo Bowne-Anderson (Substack)Sebastian Raschka (corroborated by the Kimi K3 technical report, arXiv 2607.24653)Moonshot AI (surfaced via Hacker News, 367 points / 164 comments)Interconnects VentureBeat r/LocalLLaMA arXiv NerdsChalk / r/singularity

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…