Q-Function Pretraining Barely Helps Online RL Fine-Tuning; Policy-Ensemble Bootstrapping Gives 1.26x Instead
arXiv 2607.27203 (2026-07-29, Perry Dong, Ron Polonsky, Dorsa Sadigh) systematically tests the conventional wisdom that you should pretrain the Q-function on offline data when fine-tuning on top of a pretrained policy, and finds naive Q-pretraining often provides little benefit over random initialization. The cause is a structural mismatch: the pretrained Q targets the pretrained policy's Q-function, not the one online fine-tuning converges to, and the gap survives offline value maximization. Their alternative, Initialization via Policy Ensemble, trains multiple diverse policies and bootstraps Q-learning from their pooled rollouts, averaging a 1.26x improvement over naive Q-pretraining across continuous control benchmarks.
↳ Follow the thread