A 210M-parameter model's loss stayed flat while FID kept falling

A solo developer trained a 210-million-parameter text-to-image diffusion model on one GPU in 3.5 days, then posted the internals in a training write-up on r/MachineLearning instead of the images.

The numbers matter for anyone training a diffusion model without a lab budget. Loss curves can look nearly flat while the model keeps improving in ways only an eval metric like FID or FD-DINOv2 catches.

Two learned key and value slots added to the model's cross-attention layers absorbed about 90% of attention mass at mid-noise timesteps. The end-of-sequence token, the usual place attention piles up in transformers, held onto only 4%. Around those two slots, the register vectors reached 4 to 13 times the norm of the surrounding image tokens.

Over 4.2 million training images at 256², the flow-matching loss moved only from 0.805 to 0.754. FID reached 27.0, down from 33.7 in the same stretch, and FD-DINOv2 reached 218, down from 570.

A training-time timestep shift of 2.8, tested at 20 sampling steps, produced an FID of 27.0. That beat no shift at 20 steps (FID 27.3) and came close to running the full 50 steps (FID 26.6).