PC Layer: Polynomial Weight Preconditioning for LLM Pre-Training

arXiv·medium signal

A preconditioning layer reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning to keep weight conditioning stable throughout training. Crucially, the preconditioned weights merge back into the original architecture after training, so there is zero inference overhead. Demonstrated on Llama-1B pre-training against standard transformers.