Skip to content
conftrace_
2026 EMNLP EMNLP 2026

MUDIPA: A Human-in-the-Loop AI-Assisted Platform for Discourse Graph Annotation in Multi-party Dialogues

AbstractAnnotating discourse structure in multi-party dialogues is a cognitively demanding task that requires specialized expertise, yet existing annotation tools are either fully manual or provide only limited assistance. We present MUDIPA, a web-based, human-in-the-loop platform for discourse graph annotation in multi-party dialogues, built around two pluggable AI backends: a suggestion engine that produces confidence-ranked arc proposals, and an LLM-based reasoning engine that generates on-demand natural-language rationales for candidate attachments and relation labels. In a counterbalanced, within-subjects study with eight annotators, we compare AI-assisted annotation with MUDIPA against manual annotation. The AI-assisted condition achieves higher inter-annotator agreement on relation labels, higher accuracy against the gold standard, and lower perceived workload, while requiring only a modest increase in annotation time. These findings indicate that the principal value of LLMs in discourse annotation lies in supporting human decision-making, improving both annotation quality and inter-annotator agreement relative to unassisted annotation.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Interdisciplinary

More from EMNLP 2026

All EMNLP 2026 papers →