Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
Abstract
AbstractConfidence calibration, the alignment of a model’s predicted confidence with its actual accuracy, is critical for the reliable deployment of Large Language Models (LLMs), yet remains under-explored in multilingual contexts. We conduct the first large-scale systematic study of multilingual calibration across eight model families and over 100 languages, finding that non-English languages are worse-calibrated than English (e.g., LLaMA3’s average Expected Calibration Error is 23.1% on non-English against 4.6% on English). To diagnose this, we investigate the model’s internal representations and find that the final layer, biased by English-centric training, provides a poor signal for multilingual confidence, whereas late-intermediate layers offer a better-calibrated signal. Building on this, we introduce a suite of training-free methods, including Language-Aware Confidence Ensemble (LACE), which adaptively selects an ensemble of layers for each language. Both the multilingual calibration gap and the late-intermediate peak persist beyond MCQA, in open-ended QA, chain-of-thought reasoning, and alternative MCQA scoring protocols. Our study highlights the costs of English-centric alignment and offers a path toward more globally equitable LLMs.