Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
Abstract
AbstractMultimodal large language models are increasingly given contextual evidence in heterogeneous forms: as a text passage, as a rendered image of the same text, or as both together. However, little is known about how consistently these surface forms are processed, especially when the evidence conflicts with the model’s parametric knowledge. In this work, we study modality robustness under knowledge conflict across 13 MLLMs and two datasets, and find them far from robust: (1) when a single context contradicts parametric knowledge, models tend to favor it more when presented as an image; (2) when a text and an image instead jointly conflict with parametric knowledge, the preferred source becomes essentially arbitrary, fluctuating drastically with input order, model choice, and dataset. We further show this instability affects downstream tasks like multimodal RAG and adversarial attacks: harmful instructions delivered as images, for example, raise attack success rates by up to 27 points. While conflict-aware fine-tuning partially mitigates the issue, we call for broader community attention to MLLMs’ modality robustness.