Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Characterizing and assessing robust conversational interaction: Amendments in order specification

AbstractWe assess the robustness of large language models (LLMs) on a simple task—fast-food order specification—in the face of user-initiated amendments. In amending an order, users must communicate, explicitly or implicitly, that a change is being made, the type of change, and what is being changed. We generate synthetic food orders with single amendments, controlling the type of amendment, which components are implicitly or incompletely specified, and the pragmatic inferences required to resolve them. On a dataset broadly covering the space of possible combinations, 7 flagship models average 97.7% accuracy. On a more targeted dataset with denser coverage of empirically difficult patterns, accuracy drops to 93.4%, with the hardest combinations dropping below 80% accuracy. We also test two further prompting methods: generating an order before interpreting amendments in a turn-taking interaction; and function-calling to parameterize explicit edit operations on the order itself. These somewhat improve performance (+0.2 pp and +3.2 pp respectively), but neither entirely eliminates the failure modes. We also find that asking clarification questions about implicit components improves accuracy on a subset of difficult examples (+5.4 pp), but models ask on only 11% of examples. Given the task’s deliberate simplicity—small fixed menu, a single standard order, with a single unambiguous amendment—these results illustrate the challenge of robustly performing the pragmatic inferences that real-world conversational interaction demands.

More from EMNLP 2026

All EMNLP 2026 papers →