Skip to content
conftrace_
2026 EMNLP EMNLP 2026

InterventionBench: Can LLM Agents Adapt to Mid-task User Interventions?

AbstractLLM agents are increasingly deployed on long-horizon tasks, where a user often revises the goal partway through rather than stating it once at the start. Adapting an ongoing trajectory to such a mid-task intervention is a basic requirement for agents that work with a user, yet existing agentic benchmarks evaluate agents under a goal that stays fixed for the whole task. To address this gap, we introduce InterventionBench, a benchmark that measures how well an agent adapts to a mid-task intervention. Built on AppWorld and WebArena, InterventionBench inserts a user message that revises the goal partway through each task, and defines the revision along three operations and three components of the goal, forming nine intervention types that locate which kind of revision an agent fails on. The released benchmark contains 2,250 intervention instances. We evaluate twelve LLM agents and find that every intervention type degrades performance. Add interventions, which extend the goal rather than replace it, are consistently the hardest. These results establish adapting to a mid-task intervention as a substantial and open challenge for current LLM agents. Our code and data are available at https://github.com/jhn25/InterventionBench.

More from EMNLP 2026

All EMNLP 2026 papers →