Object Presence Is Not Relation Evidence: Relation-Grounded Faithful Reasoning for Multimodal Large Language Models
Abstract
AbstractRecent Multimodal Large Language Models (MLLMs) excel in precise entity recognition and powerful reasoning, significantly boosting performance and efficiency in complex vision-language tasks. Despite these gains, this advanced reasoning process introduces a subtle yet critical vulnerability. We identify an underexplored relational reasoning failure mode in which models over-prioritize semantic coherence, thereby inferring ungrounded interactions from mere object entities. We term this phenomenon Relation Evidence Amnesia (REA), wherein MLLMs correctly localize individual entities but fail to enforce true visual dependencies as the generative process becomes hijacked by semantic rationalization. To mitigate REA, we propose Relation-Grounded Faithful Reasoning (RGFR), an inference-time decoding intervention that operates without modifying target model parameters. Adopting a decoupled verification philosophy, RGFR employs a semantic-driven soft-gating mechanism to simultaneously incorporate explicit entity constraints and extract counterfactual causal evidence, followed by a Unified Schema-Aware Rectifier that rigorously mediates these visual-causal constraints during the decoding process. Extensive experiments across relation benchmarks demonstrate that RGFR improves relation recognition accuracy by over 8.1% compared to baselines, while largely preserving task utility. These results highlight the critical role of decoupled, evidence-grounded rectification in achieving faithful relational reasoning for multimodal models.