Do LVLMs Know When They Are Wrong? Trajectory-Based Correctness Estimation from a Single Forward Pass
Abstract
AbstractLarge vision-language models (LVLMs) exhibit strong multimodal reasoning capabilities, yet reliably estimating whether a generated response is correct remains a fundamental challenge. Existing confidence estimation methods commonly rely on repeated querying, self-consistency sampling, or layer-wise probing, which introduce additional inference overhead, self-preference bias, or costly full-depth analysis. In this work, we propose a trajectory-based framework for estimating LVLM response correctness from a single forward generation pass. Instead of treating internal representations as isolated layer-wise signals, we model response generation as a structured trajectory through the LVLM, capturing the evolution of visual evidence in the vision encoder, geometric transformations induced by cross-modal projection, and probability dynamics within the language decoder. Our key insight is that response correctness is encoded in the geometry and dynamics of these component-specific trajectories. Based on this observation, we derive lightweight trajectory descriptors for each major LVLM module and integrate them through a calibrated correctness estimator trained with as few as 500 labeled examples. At test time, our method requires no additional inference passes and no ground-truth labels. Extensive experiments across multiple LVLM architectures and benchmark datasets demonstrate consistent improvements over existing baselines, including up to 16.44 higher AUROC on Qwen-2B over VisWiz and 15.61 higher AUPR on LLaVA-Next-7B over ScienceQA compared to the strongest competing methods.