TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents
Abstract
AbstractLarge language model (LLM) agents increasingly act through files, tools, memory, retrieved content, and other agents, which can trigger safety failures during execution. Existing benchmarks mostly focus on endpoint outcome evaluation or model-assisted moderation of pre-constructed trajectories, leaving limited evidence about where risks enter and how they propagate. To address this issue, we introduce TraceProbe, an execution-based benchmark for agent safety diagnosis. TraceProbe places agents in executable sandbox scenarios, scores endpoint and latent rule violations based on normalized traces and structured evidence, and replays recorded tool actions for step-level risk localization. Our benchmark contains 155 scenarios across three risk-source categories: non-adversarial failures, internal authority compromise, and external attacks. Across 12 contemporary LLMs, the mean attack success rate (ASR) is 0.347, indicating substantial residual risk, especially under internal authority compromise. Our analysis also demonstrates that endpoint-safe runs can contain unsafe intermediate actions, and the actual execution paths of agents may differ from the intended hazards of the testing scenarios. The TraceProbe benchmark and our findings support a more actionable form of agent safety evaluation: beyond assessing ASR or final outcomes, evaluators can also ask what went wrong, where it occurred, and where an intervention could be placed.