Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Can LLMs Reliably Identify Causal Conclusions in Scientific Literature? A Multidisciplinary Evaluation

AbstractLarge language models (LLMs) hold promise for automating causal knowledge extraction, a fundamental component of scientific discovery, yet their ability to reliably identify causal claims in scientific literature remains poorly understood. Using 1,000 manually annotated abstracts spanning 20 disciplines, we evaluate eight frontier LLMs under four prompting strategies on a causal conclusion identification task. More elaborate prompting strategies do not necessarily improve performance, while open-weight models achieve competitive performance relative to proprietary models. LLMs systematically produce false-positive causal predictions on non-causal abstracts, with GPT-5 showing the most extreme bias, and Few-Shot prompting provides the most consistent calibration benefit. Performance varies substantially across disciplines with stronger and more stable results in biomedical fields. Our findings provide practical insights for using LLMs in scientific causal analysis and highlight calibration and domain variation as key challenges for advancing causal reasoning in scholarly text.

❓ The Questioner
🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →