From Pixels to Tokens: Adaptive LLM Decoding and Reliability Analysis with Otsu Thresholding
Abstract
AbstractThis paper investigates the relationship between generation-time uncertainty and output reliability in probabilistic Large Language Model (LLM) decoding. Specifically, we assess the correlation between several factuality, quality and certainty metrics to infer existing patterns across state-of-the-art decoding strategies, diverse tasks and instruction-tuned models. We also propose a novel, image processing-inspired decoding approach, Otsu Thresholding, that performs adaptive, parameter-free threshold selection directly from the next-token predictive confidence distribution. The proposed Otsu-based variants achieve competitive quality and factuality across tasks and models, while exhibiting concentrated predictive distributions comparable to strong truncation-based baselines. Furthermore, we employ a selective filtering setup to analyze the impact of removing highly uncertain instances on the quality and factuality metrics, ultimately revealing that filtering based on averaged token-level uncertainty metrics can be practically useful for identifying less reliable generations, with filtering frequently improving the quality or factuality of the retained outputs.