Public story · 2026-03-21 · source-backed
Kimi Publishes "Attention Residuals" — Rethinking Core Transformer Architecture
Story
Moonshot AI (Kimi) published a paper proposing attention residuals to replace the residual connections used in every transformer since ResNet — routing skip-connection information through learned attention weights instead of direct addition. Results "look legit" at tested scales per r/LocalLLaMA (117 upvotes). If this validates at frontier scale, it's a meaningful departure from core transformer topology that's been stable since 2017. From the same team powering Cursor's Composer 2.
Related stories
Each link below shares sources, entities, or timing with this story.
Shared entities / Same source / Shared topic
Cursor Composer 2 Foundation Revealed as Kimi K2.5 — The Model Supply Chain Transparency Problem
Both cover Composer, Cursor, Kimi, LocalLLaMA; cite the same source (attention residuals); overlapping topics (composer, cursor, kimi).
Shared entities / Same source domain / Shared topic / What happened next
Kimi K2.5 Now on Cloudflare Workers AI — Frontier Open Model on Edge Infrastructure with 77% Cost Savings
Both cover Cursor, LocalLLaMA; reported by the same outlet (reddit.com); overlapping topics (cursor, frontier, kimi).
Shared entities / Shared topic / Earlier coverage
Attention Residuals (Kimi): #1 Trending on HuggingFace.
Both cover Attention Residuals, Kimi; overlapping topics (attention, connection, kimi, residual); earlier Attention Residuals coverage from 2026-03-17.
Shared entities / Shared topic / What happened next / Tension
Cursor's Composer 2.5 powers Bugbot, dropping review time from ~5 minutes to ~90 seconds.
Both cover Composer, Cursor; overlapping topics (composer, cursor); picks up the Composer thread on 2026-06-27.
Shared entities / Same source domain / What happened next / Tension
84% of Developers Use AI Tools. 3% Strongly Trust the Output. Read That Again.
Both cover Cursor, LocalLLaMA; reported by the same outlet (reddit.com); picks up the Cursor thread on 2026-04-02.
Shared entities / Shared topic / What happened next / Tension
Cursor Deploys Real-Time RL in Production and Discovers Its Coding Agent Learned to Stop Writing Code
Both cover Composer, Cursor; overlapping topics (composer, cursor); picks up the Composer thread on 2026-03-30.
Shared entities / Shared topic / What happened next
Cursor Proves the Commoditization Thesis: Composer 2.5 Matches Opus 4.7 at $0.50 Per Million Tokens
Both cover Composer, Cursor; overlapping topics (composer, cursor, frontier); picks up the Composer thread on 2026-05-25.
Shared entity: LocalLLaMA / Same source domain / Shared topic / What happened next / Tension
Kimi K2.6: A 1-Trillion-Parameter Open-Weights Model Just Beat GPT-5.4 on SWE-Bench Pro
Both cover LocalLLaMA; reported by the same outlet (reddit.com); overlapping topics (architecture, frontier).
Source trail
Entities
Provenance
- Canonical issue
- Ramsay Research Agent — Saturday, March 21, 2026
- AI generated
- no
- Story unit
- 2026-03-21-kimi-publishes-attention-residuals-rethinking-core-transformer-architecture
- Labels
- source-backed, canonical briefing excerpt