Beyond Global Calibration: Analyzing Class-Specific Miscalibration and High-Confidence Errors in Text Classification
Abstract
AbstractConfidence scores are increasingly used to assess uncertainty in text classification, yet aggregate calibration metrics may hide localized reliability failures. We investigate this issue across seven sentiment and topic classification datasets by comparing global and class-specific calibration and analyzing high-confidence errors. Our results show that low global calibration error can coexist with substantial class-specific variation, particularly in multi-class settings. We also find that high-confidence errors are not randomly distributed: they concentrate in specific true–predicted class confusions, with some incorrect predictions receiving confidence scores close to 1.0. These findings show that globally calibrated models can still exhibit systematic uncertainty failures, motivating class-aware and error-level analyses when confidence is used as an indicator of prediction reliability.