Information Abundance Paradox: Training on Longer Contexts Past an Intermediate Optimum Makes Models Worse at Closed-Book Recall

arXiv 2608.12218·high signalXBlueskyLinkedInCopy link

The paper challenges the assumption that longer training contexts are monotonically good. In pretraining on long documents, increasing the context window improves language modeling, NLU, and closed-book MCQA only up to an intermediate optimum, after which performance consistently declines; in SFT, more task-relevant train-time context helps when supporting context is present but reduces robustness when context is absent or misleading at test time. Mechanistically, informative context shifts gradient pressure away from feed-forward networks (linked to parametric knowledge) toward attention, and causal interventions confirm this increases inference-time context reliance.