Rescaling the Backward Signal Lets Recurrent Models Extrapolate to 128x Their Training Length

arXiv 2609.09157·medium signalXBlueskyLinkedInCopy link

Recurrent models trained with backpropagation through time typically fail beyond their training horizon, and the paper argues vanishing or exploding gradients alone do not explain this, since dense per-token losses can still train a shared recurrent rule under severe decay. The authors instead target state credit, the signal by which future losses reach earlier recurrent states, and propose Credit Stabilization through Time, which locally rescales the state-credit signal during backward propagation to stabilize its norm without rotating the corrected component and without changing the forward computation. Specialized separately for synthetic and real-data credit dynamics, it improves performance beyond the training horizon with gains observed at up to 128x the training length.