Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Credal Ensembles for Uncertainty Quantification in LLMs

AbstractWe consider the problem of uncertainty quantification in LLMs from a black-box perspective. To effectively disentangle aleatoric and epistemic uncertainty, we use an ensemble method that produces responses paired with so-called credal sets of distributions. We combine different ensembling techniques, based on temperature variation, multiple models, paraphrasing, and permutations, with semantic-similarity and verbalized elicitation methods, resulting in new, computationally cheap measures of uncertainty. A benchmark on closed-ended (BoolQ and CommonsenseQA) and open-ended (AmbigQA) questions shows good correlation with predictive accuracy. Notably, when considering ambiguous tasks, our analysis reveals a tendency of LLMs to focus on a single interpretation, an important flaw in current LLM-based solutions.

More from EMNLP 2026

All EMNLP 2026 papers →