Data Temporality in Pre-Training: Shuffled Corpora Freeze LLM Knowledge at Train Time
arXiv·medium signal
Study shows that standard shuffled-corpus pre-training produces models whose knowledge is frozen at train time, unable to reason about when facts change. Temporal ordering during pre-training significantly improves the model's ability to handle evolving knowledge, directly relevant to reducing hallucination about outdated facts. Practical finding for anyone designing pre-training data pipelines.