Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Causal Head Gating: Separating Visual Evidence and Knowledge Priors for Hallucination Mitigation in Multimodal LLMs

AbstractMultimodal large language models excel at vision-language tasks but suffer from hallucinations that undermine their reliability. Existing mitigation methods apply uniform heuristics at the output or attention level, overlooking the heterogeneous causal roles of attention heads and their dynamic behaviors during inference. We revisit hallucination from a causal perspective and identify it as competition between two pathways: a direct path from visual evidence to outputs, and an indirect path dominated by knowledge priors. When visual evidence weakens, MLLMs increasingly rely on parametric knowledge priors—such as spurious correlations learned during pretraining. This over-reliance causes attention heads responsible for visual grounding to drift toward language-driven patterns, resulting in generations consistent with the language model but unfaithful to the image. We propose Causal Head Gating, an inference-time framework that separates visual evidence from knowledge priors by modeling their causal contributions at the attention-head level. Under this framework, we introduce CHERI (Causal Head Evidence Reweighting and Intervention), which identifies attention heads whose visual contributions weaken under hallucination and dynamically restores their influence. Specifically, CHERI reinforces heads that propagate visual evidence while suppressing those dominated by knowledge priors, strengthening the direct causal effect of vision on generation and attenuating spurious indirect effects. Experiments across multiple benchmarks show that CHERI consistently reduces hallucination rates across diverse task formats while preserving overall generation quality and task performance.

More from EMNLP 2026

All EMNLP 2026 papers →