Simulator Collapse: RL Against a Single LLM User-Simulator Overfits, and Population Co-Training Recovers 14% of Held-Out Success

arXiv 2608.12253·high signalXBlueskyLinkedInCopy link

Multi-agent RL for human-AI interaction almost always trains a policy against one LLM simulating the user. The authors show this systematically fails to generalize because the simulator itself is mode-collapsed, so the policy learns to exploit its dominant mode and transfers poorly to unseen simulators and real humans. Verbalized Sampling recovers up to 9% held-out success over single-simulator RL and Population Co-Training reaches 14%, with a human study confirming a similar gain on real users; they release SCOPE, an open-source Population Co-Training framework.