Structural Uncertainty from Self-Preference in Black-Box Language Models
Abstract
AbstractLarge language models can be confidently wrong, making uncertainty estimation especially important in black-box settings where token probabilities are unavailable. Existing post-hoc methods estimate uncertainty from output dispersion, treating sampled responses largely as exchangeable. We propose structural uncertainty, an observable signal derived from self-preference-induced ranking stability: given multiple sampled candidates, we elicit sparse pairwise self-preferences, aggregate them into ranking distributions via Bradley–Terry modeling with PageRank, and decompose uncertainty into two complementary entropy-based components—across-trial ranking instability and within-trial candidate ambiguity. Across five LLMs and eight benchmarks, self-preference-derived structural signals can provide information complementary to output dispersion, and their combination improves selective prediction on several reasoning and knowledge tasks. The two components relate differently to accuracy: on math reasoning, within-trial ambiguity can be positively correlated with correctness—consistent with settings where multiple plausible candidates remain competitive under self-preference ranking—whereas on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime where preference-based uncertainty is uninformative. Structural uncertainty is not a universal replacement for dispersion but a regime-sensitive complementary signal.