Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs

AbstractState-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines.

More from EMNLP 2026

All EMNLP 2026 papers →