Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory
Abstract
AbstractDiagnosing failure patterns in Deep Research Agents (DRAs) remains a critical challenge. Existing benchmarks predominantly rely on end-to-end evaluation, obscuring intermediate hallucinations that accumulate throughout the research trajectory. To bridge this gap, we propose a shift from outcome-based to **process-aware evaluation** by auditing hallucinations in the full *plan-search-summarize* trajectory. We introduce the **PING Taxonomy**, which categorizes DRA hallucinations into four complementary types: **P**ropagation, **I**ntent, **N**oise-induced, and **G**rounding. We further instantiate this taxonomy into a fine-grained evaluation framework that decomposes trajectories into atomic actions, claims, and sub-queries for rigorous verification, and we validate its reliability on standard fact-checking benchmarks and human-reviewed trajectories. Leveraging this framework to isolate 100 hallucination-prone tasks, including adversarial scenarios, we curate **DeepHalluBench**. Experiments on six representative DRAs show that, on our hallucination-prone stress-test set, all evaluated systems still exhibit non-negligible reliability gaps. Furthermore, our diagnostic analysis traces these failures to systemic deficits, especially hallucination propagation and cognitive biases, providing actionable insights for future architectural optimization. Code and data are available at https://github.com/yuhao-zhan/DeepHalluBench.