Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Beyond Calibration: Temporal LLM Confidence Assessment Across Languages and Local Settings

AbstractLarge language models are increasingly expected to estimate their own uncertainty, with confidence signals being used to guide retrieval, adaptive compute allocation, routing, and selective abstention. In this work, we investigate whether confidence signals measured at different temporal stages of problem solving encode distinct forms of epistemic information, and whether such signals remain reliable beyond English-centric settings. Inspired by classical meta-memory theory, we distinguish between anticipatory pre-solve confidence (Feeling-of-Knowing, FOK) and retrospective post-solve confidence (Judgement-of-Learning, JOL). We evaluate these signals across frontier LLMs, multiple reasoning domains, translated Marathi benchmarks, and a newly constructed benchmark of genuinely local Marathi factual questions. Across settings, the cheaper pre-solve signal remains weakly calibrated, while post-solve confidence generally provides stronger epistemic alignment despite additional inference cost. We further find that confidence behaviour transfers surprisingly well under direct language translation, with calibration trends largely preserved between English and Marathi. In contrast, confidence reliability degrades substantially on local Marathi knowledge, suggesting that linguistic transfer alone is insufficient to capture the challenges posed by region-specific information needs. Finally, hallucinated responses maintain persistently elevated confidence even after answer generation, highlighting important limitations of current LLM self-assessment mechanisms.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →