Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Simulating Multimodal Multiparty Student Interactions with Large Language Models for Training Adaptive Pedagogical Agents

AbstractMultiparty tutoring policies must regulate turn-taking and social attention, yet collecting the interaction data needed for reinforcement learning is expensive in educational settings. We introduce an LLM-based user simulator for triadic robot–student dialogue that represents gaze and gesture as inline stage directions. Combined with phase-tagged prompting (which delineates conversational structure) and per-turn persona reinforcement, this design allows the LLMs to generate interactions that preserve key discourse structure and controllable participation dynamics. We validate six simulator variants against a real student–robot corpus (12 student pairs) using metrics for semantic similarity, cue obedience, and persona consistency. While the simulator reaches strong semantic agreement with real transcripts, it also reveals a measurable sim-to-real style gap (in verbosity and grammatical complexity). The simulator enables the creation of an RL environment where the tutor selects factorized gaze and reference actions, learning to interact with simulated students. We compare online training (APPO) to offline learning (DQN). Off-policy evaluation on real logs shows that offline policies trained purely from synthetic data exceed online-trained policies under a doubly-robust estimator, and a verbal-only ablation yields unstable OPE due to heavy-tailed importance weights. These results support a scalable workflow for prototyping socially grounded multiparty tutoring policies while relaying where real-world coverage and OPE reliability remain bottlenecks.

More from EMNLP 2026

All EMNLP 2026 papers →