Vector Policy Optimization: Training LLMs for Diversity Improves AlphaEvolve-Style Inference Search

arXiv·medium signal

Researchers propose Vector Policy Optimization, which trains LLMs against multiple reward functions simultaneously instead of a single scalar reward. Standard post-training collapses response diversity, hurting inference-time search procedures like AlphaEvolve that need varied rollouts. VPO produces higher-entropy distributions that yield better search performance. Practical for anyone building inference-scaling systems that rely on diverse candidate generation.