Optimal Representation Size: Theory Explains When Wider Pretraining Embeddings Hurt Downstream Performance

arXiv·low signal

High-dimensional analysis of pretraining and linear probing reveals that there is an optimal representation dimension — going wider isn't always better. The work provides theoretical grounding for a practical observation many practitioners have encountered: larger embedding dimensions can hurt generalization from limited labeled data. Useful for teams sizing representation layers in foundation model fine-tuning.