Skip to content
conftrace_
2026 EMNLP EMNLP 2026

CausalPrism: Component-Level Evaluation of Intervention-Based Causal Reasoning in Large Language Models

AbstractEvaluating causal reasoning in large language models (LLMs) is complicated by benchmarks that conflate distinct reasoning operations into aggregate scores. We propose CausalPrism, an evaluation framework grounded in structural causal model semantics and cognitive models of causal inference, which decomposes intervention-based causal reasoning into three functional primitives—Causal Prior, Forward State Tracing, and Belief Updating—plus a consistency primitive, Agency Sensitivity. Primitives are instantiated in directed acyclic graphs and evaluated across three narrative domains of varying semantic richness. Across 14 open-source and frontier LLMs, Belief Updating is the dominant bottleneck: the best model reaches only 0.613 accuracy, while GPT-5.2 and GPT-5-mini score 0.531 and 0.517 despite near-ceiling performance on Causal Prior and Forward State Tracing. Prior exhibits systematic domain sensitivity and over-rejection. Agency framing effects are family-dependent for weaker models. We argue against treating causal reasoning as a unitary capability and motivate targeted evaluation and training along multiple primitive axes.

More from EMNLP 2026

All EMNLP 2026 papers →