Kimi Attention Residuals: Learned Per-Layer Weights Replace Fixed Unit Accumulation in Transformer Residuals
arXiv·medium signal
The Kimi team published arXiv:2603.15031 proposing learned per-layer scalar weights for residual connections, replacing the standard fixed weight-1.0 accumulation in PreNorm transformers. Standard LLMs sum all layer outputs equally; learned weights allow the model to selectively weight each layer's contribution. Potential drop-in architectural improvement for any transformer stack.