ThermoDPO Names Manifold Drift as a Root Cause of Reward Hacking in Flow Preference Optimization

arXiv 2608.20011·medium signalXBlueskyLinkedInCopy link

Extending preference optimization to flow matching is unstable because reward-driven updates modify transport trajectories with nothing anchoring terminal samples to the pretrained data manifold. The authors formalize this as manifold drift, proving that optimal flow matching recovers the terminal data distribution while a preference update leaves the pretrained manifold whenever its induced terminal displacement has a nonzero normal component. Their remedy, ThermoDPO, is a temperature-controlled objective anchoring pairwise preference optimization on preferred samples; the weighted variant reaches a StrictScore of 0.899 against 0.629 for FlowDPO and 0.857 for FlowDPO+RFT, and on SD3.5-M at CFG 4.5 improves OCR by 47.5% and a four-metric average by 16.0%.