Skip to content
conftrace_
2026 EMNLP EMNLP 2026

INSIGHT: Structured Multimodal Knowledge Modeling for Vision-Guided Task-Oriented Dialogue

AbstractIn real-world task-oriented dialogue (TOD), users often express their intents through both text and images, with images providing essential cues for fine-grained requirements. Addressing such scenarios demands effective cross-modal semantic modeling and precise knowledge retrieval. However, existing TOD methods primarily focus on text and assume one-to-one attribute mappings, failing to reflect the multimodal nature of real-world information. To overcome these limitations, this paper proposes an end-to-end vision-guided retrieval-generation framework named INSIGHT. It introduces a progressive dynamic fusion mechanism that performs layer-wise cross-modal interaction to achieve robust multimodal dialogue understanding. In addition, a graph-structured entity modeling strategy explicitly models one-to-many image attribute mappings through distributed latent semantic learning, thereby preserving visual diversity. Finally, knowledge retrieval and response generation are jointly optimized in an end-to-end manner. To evaluate real-world scenarios, we construct VISTA, a new multimodal TOD dataset that features image-driven multi-turn dialogues and supports one-to-many entity attribute mappings. Experimental results on two multimodal datasets demonstrate that INSIGHT consistently outperforms existing methods across various backbone models.

More from EMNLP 2026

All EMNLP 2026 papers →