Skip to content
conftrace_
2026 EMNLP EMNLP 2026

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

AbstractLarge language models are reshaping e-commerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce RealWorldShop, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and action-controlled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose RealShop_Agent, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that RealShop_Agent consistently outperforms strong baselines on RealWorldShop.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →