Trust-Aware Multi-Axis Reliability Auditing for NLP Classifiers: A Theoretical and Empirical Framework
Abstract
AbstractAccuracy remains the dominant criterion for selecting NLP classifiers, yet it fails to characterize the operational reliability required for deployment under class imbalance, distribution shift, and demographic heterogeneity. We introduce TNMM++ (Trustworthy NLP Multi-axis Monitoring, extended), a theoretically grounded reliability audit that evaluates classifiers along seven complementary axes, utility, calibration, robustness, safety, fairness, selective prediction, and uncertainty quantification, and formalises model selection as constrained multi-objective optimization. Beyond metric aggregation, TNMM++ contributes six algorithmic results: a reliability tensor, a Pareto convex-hull theorem, variance-adaptive weighting, a boundary-predictor criterion with exclusion rule for majority-class collapse, a Bayesian risk decomposition, and the TAMS algorithm with optimality bound. Evaluating eight lightweight Transformer encoders (MiniLM, ELECTRA, mBERT, DistilBERT, MobileBERT, ALBERT, RoBERTa, XLM-R) and excluding models that failed to learn the task, we find that properly trained mid-tier models exhibit balanced utility and calibration superior to collapsed high-accuracy counterparts. TrustScore identifies mBERT (0.6685), ELECTRA (0.6630), and DistilBERT (0.6627) as deployment-reliable. Sensitivity analysis under 1,000 Dirichlet weight perturbations confirms ranking stability in 92.4% of configurations, and multi-seed evaluation (5 seeds) demonstrates conclusions are not seed artifacts. Our findings establish that single-metric selection is statistically unreliable and motivate multi-axis reliability auditing as the default protocol for trustworthy NLP deployment.