Skip to content
conftrace_
2026 EMNLP EMNLP 2026

SMART: Sink-based Modality-Aware Redistribution of Transformer Attention

AbstractMultimodal Large Language Models (MLLMs) often exhibit modality bias, where predictions over-rely on one modality while underutilizing others. However, the mechanism underlying modality bias remains poorly understood. In this work, we first introduce Banana-Counting, a controlled diagnostic dataset for measuring modality preference under cross-modal conflicts. Evaluations across diverse MLLMs reveal pervasive modality imbalance. Then, through attention analysis, we show that modality bias is strongly associated with imbalanced cross-modal attention allocation and modality-specific attention sinks that absorb redundant attention mass. Motivated by these findings, we propose SMART, an inference-time method that mitigates modality bias by redistributing redundant sink attention toward under-attended modalities. SMART consistently improves multimodal balance across model families. On Banana-Counting, it reduces the modality gap from 27.73 to 0.66 on LLaVA-v1.5-7B while improving balanced multimodal accuracy. Moreover, SMART generalizes effectively to downstream multimodal reasoning tasks without retraining or parameter updates.

More from EMNLP 2026

All EMNLP 2026 papers →