Moonshot AI Introduces Attention Residuals (AttnRes) — Replaces Fixed Residual Connections with Learned Depth-Wise Attention
NerdsChalk / r/singularity·medium signal
Kimi's team announced Attention Residuals on March 16: instead of uniformly adding each layer's output to a shared hidden state, AttnRes computes weighted combinations over all previous layers — treating model depth like a sequence dimension. This lets each layer selectively retrieve useful representations from earlier points, addressing the 'equal blending' limitation of standard residuals. Targets improved scaling stability, information preservation in deep networks, and context retrieval in large-scale MoE architectures.