MEETINGQA2: A Generative Speaker-Centric Benchmark for Multi-Party Dialogue Question Answering
Abstract
AbstractMulti-party Dialogue Question Answering (MDQA) requires the machine to answer questions based on a conversation with multiple interlocutors. Existing benchmarks are largely extractive, leaving positional shortcuts and providing limited evidence about whether models can synthesize distributed evidence into speaker-grounded answers. We introduce MEETINGQA2, the first generative MDQA benchmark, derived from MEETINGQA with reformulated questions and speaker-centric summary answers. It contains 1,033 QA pairs, including 108 unanswerable questions. Beyond standard overlap metrics, we introduce attribution accuracy and factual correctness. Experiments show that strong LLMs perform consistently worse on MEETINGQA2 than on MEETINGQA, with especially large drops in attribution accuracy. The results indicate persistent weaknesses in speaker attribution, factual grounding, and long-context dialogue understanding.