Deriving Necessary Validity Requirements for Evaluating the Consistency of LLM Responses
Abstract
AbstractThe probabilistic nature of large language models (LLMs) has regularly invited inquiries into their consistency with regard to repeated sampling, input variations, and conceptually related prompts. However, unlike accuracy benchmarks, consistency evaluations cannot rely on standardized measurement tools. We conduct a comprehensive survey of current methodologies evaluating language model consistency across 85 peer-reviewed publications, organizing them into a novel three-level taxonomy: (1) sampling, (2) semantic, and (3) relational consistency. Throughout this process, we identify three fundamental threats to the construct validity of LLM consistency measurements: Valid evaluations must disentangle intrinsic consistency from correctness, probabilistic uncertainty, and artifacts of constrained generation. Our systematic analysis shows that none of the 85 publications accounts for all three factors jointly, allowing construct-irrelevant variance to obscure our understanding of consistency.