2026
EMNLP
EMNLP 2026
Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs
Abstract
AbstractState-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines.