Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Evaluating LLM Agents Beyond Local Edit Validity

AbstractEvaluations of sequential large language model (LLM) agents often report local action validity or final success. Delayed constructive tasks additionally require recoverability, meaning each accepted edit must leave the partial artifact completable within the remaining budget. We introduce a measurement protocol across Word Ladder, an alloy composition proxy, and the fitness landscape of the B1 domain of protein G (GB1). We call an episode surface clean when every action is well formed and locally admissible. Across six models, Word Ladder exposes interface failure with 36.1% local validity. Alloy and GB1 episodes are often surface clean, yet only 16.0% and 6.8% of them succeed. Exact enumeration of 625 GB1 variants over budgets from zero to six decomposes failure. Among surface clean episodes, 51.4% cross a recoverability boundary, while 14.1% of the remaining episodes succeed. Bounded composition audits and intervention checks connect bottlenecks to repair classes. These results establish recoverability as a decomposable and auditable evaluation axis that informs interventions for sequential agents.

More from EMNLP 2026

All EMNLP 2026 papers →