LittleLearner Trained LLMs on an 88B-Token K–5 Corpus and Found Post-Training Can't Break the Pretraining Ceiling

Max Planck / ELLIS / Hacker News·medium signalXBlueskyLinkedInCopy link

A seven-author team from Max Planck Institute for Intelligent Systems, ELLIS Tübingen, and ETH Zürich (led by Fanfei Li and Jana Zeller) built LittleCurriculum — 88B tokens distilled from FineWeb-Edu through a five-stage Common Core K–5 filter — and trained 0.6B, 1.3B, and 5B models from scratch, each with a matched unfiltered control. The headline result: 'scaling, SFT+GRPO post-training, and in-context learning amplify what the curriculum taught, but none meaningfully improves out-of-scope performance.' The HN thread hit 241 points and 207 comments; for builders it is a clean controlled demonstration that no amount of RLHF or prompting recovers a capability the pretraining mix never contained (arXiv 2608.13545).