Skip to content
conftrace_
2026 EMNLP EMNLP 2026

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

AbstractMulti-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, a LLM policy trained against it overfits to narrow strategies that exploit the simulator’s mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two solutions. The inference-time solution, Verbalized Sampling, broadens the simulator’s behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator’s mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, 𝜏2-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users in both. Both solutions preserve the policy diversity that collapses under single-simulator RL. Broadly, our results %challenge the common practice of a frozen simulator and suggest that the diversity of the training environment is critical to the generalization of multi-agent RL to real users.

More from EMNLP 2026

All EMNLP 2026 papers →