Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Evaluating LLM-generated Explanatory Dialogue Turns through Dialogue Completion

AbstractSynthetic explanatory dialogues hold promise for applications from tutoring systems to model rationalization, yet their quality is often deemed inferior to human-authored conversations. This study uses dialogue completion as a lens to investigate the nuanced differences between human-spoken, human-written, and synthetic explanatory dialogue. By evaluating LLM-generated turns against original turns using the FED and IXQuisite test suites, we uncover a significant bias in the FED metric towards written language, which may penalize the interactivity of spoken dialogue. We find that three popular LLMs (Mistral, Llama, Claude) replicate this pattern, excelling in fluency, but struggling with conversational engagement, particularly on spoken dialogue transcripts. More importantly, we demonstrate that these models can significantly improve their performance through a zero-shot, iterative self-refinement method. By providing feedback based on their specific weaknesses, we show that LLMs can iteratively adapt their outputs to better resemble the interactive qualities of human dialogue.

More from EMNLP 2026

All EMNLP 2026 papers →