Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Procedural Blind Spots in Agent Evaluation: A Survey, Coverage Analysis, and New Missions for NLP

AbstractLarge Language Model (LLM) agents increasingly operate in multi-step, multi-turn settings that combine user interaction, tool use, state changes, and communication. Most agent evaluation still emphasizes task completion, while procedure-aware checks remain fragmented. We conduct a systematic survey and introduce ℱ*, a systematic organizing map that maps scattered procedural checks into a shared audit space spanning integrity relations and quality dimensions. Applying this framework to recent agent-evaluation work reveals three procedural blind spots: most work remains outcome-focused or procedurally under-specified, procedure-aware checks mostly remain local spot-checks, and communication grounding lags action compliance. To close these gaps, we propose four new Natural Language Processing (NLP) missions around communication grounding, constraint semantics, reasoning faithfulness, and procedural equity. This survey provides a comprehensive overview of procedure-aware agent evaluation, offering a common vocabulary and coverage map that clarify what current benchmarks can certify and provide clear guidelines for future research.

More from EMNLP 2026

All EMNLP 2026 papers →