Pixel-Space Diffusion Can Match Latent Models and Run 3.18-4.75x Faster — If You Pretrain in Latent Space First

arXiv 2608.16887 (HuggingFace Daily Papers)·medium signalXBlueskyLinkedInCopy link

A large empirical study finds direct large-scale pretraining in pixel space converges substantially more slowly than latent-space pretraining, which motivates a latent-to-pixel recipe: acquire generative priors cheaply in latent space, then transition to pixel space during post-training. The authors systematically sweep the transition's design choices — weight initialization, data composition, prediction target, decoder architecture, and noise schedule — and land on a recipe whose pixel-space models match or beat latent-space counterparts while delivering 3.18x to 4.75x end-to-end inference speedups. That removes the main practical objection to dropping the VAE from text-to-image stacks.