Skip to content
conftrace_
2026 EMNLP EMNLP 2026

DisasterBench: Benchmarking LLM Planning under Typed Tool Interface Constraints

AbstractDisasters cause severe societal impacts, demanding rapid coordination of heterogeneous AI tools, from satellite analysis to flood prediction and damage assessment, into coherent multi-step workflows. As LLMs increasingly serve as orchestrators of such pipelines, effective coordination requires more than selecting semantically plausible tools: LLMs must generate interface-valid workflows with correct parameter binding and dependency propagation. We introduce DisasterBench, a benchmark for evaluating structured multi-agent planning over semantically similar but operationally distinct disaster-response tools. To enable step-level failure attribution, we further propose First-Point-of-Failure (FPoF), which localizes the earliest root cause in a predicted workflow, separating primary errors from downstream cascading effects. Our evaluation reveals three findings: planning method effectiveness depends strongly on model capacity; tool mismatch and parameter-binding errors dominate first failures, revealing semantic grounding and interface consistency as distinct bottlenecks; and verbose intermediate reasoning can create instruction clash with structured output requirements, disrupting plan generation. Together, these findings highlight a fundamental gap between semantic reasoning and interface-constrained workflow planning, underscoring the need for planning frameworks that jointly model semantic intent, execution constraints, and workflow consistency. All code, data, and evaluation resources are available at the https://github.com/TamuChen18/DisasterBench_Open.

More from EMNLP 2026

All EMNLP 2026 papers →