Beyond Visual Neglect: Context-Preference Activation Steering for Mitigating MLLM Hallucinations
Abstract
AbstractObject hallucination remains a primary obstacle to the reliable deployment of Multimodal Large Language Models (MLLMs). Current inference-time mitigation methods mainly assume that hallucinations stem from visual neglect and therefore steer models toward greater visual reliance. In contrast, our systematic interventions on multiple MLLMs show that increasing visual reliance may exacerbate hallucinations on some models, whereas reducing it may mitigate them. This result suggests that visual neglect alone provides an incomplete explanation of hallucination. We characterize the image as a context that competes with both the model’s parametric knowledge and its textual context. Based on this characterization, we propose a training-free framework, Context-Preference Activation Steering (CAS). CAS extracts two Context Preference Vectors (CPVs), designed to capture different context-preference relationships, from two small sets of conflict samples and injects signed residuals into the MLP-sublayer outputs before residual addition at four mid-early layers in a single forward pass. Experiments show that CAS substantially mitigates object hallucinations with negligible decoding overhead while maintaining F1, lexical diversity, and repetition rates close to those of the base models. Code and data are available at https://github.com/wss7/CAS.