UserBench: An Interactive Gym Environment for User-Centric Agents
Abstract
AbstractLarge Language Model agents increasingly solve tool-use tasks, yet often fail to determine what users actually want when goals are underspecified, incremental, or indirectly expressed. We introduce UserBench, a controlled Gymnasium environment for evaluating proactive preference elicitation in multi-turn, tool-grounded interactions. Instantiated in travel planning, UserBench hides user preferences behind implicit natural-language cues and requires agents to ask targeted clarification questions, search structured option databases, and select user-aligned options. Evaluating leading open- and closed-source models reveals a large gap between tool execution and user understanding: models achieve high valid-search rates but elicit only a minority of hidden preferences, and even stronger models remain far below a human oracle. Further analysis shows that many additional interaction turns are broad or non-informative rather than preference-relevant. UserBench provides a reproducible testbed for studying and training user-centric agent interaction while intentionally abstracting away full travel negotiation and infeasible-constraint recovery.