Skip to content
conftrace_
2026 EMNLP EMNLP 2026

RAVE: Re-Allocating Visual Attention in Large Multimodal Models

AbstractLarge multimodal models (LMMs) inherit the self-attention mechanism of pretrained language backbones, yet standard attention can exhibit suboptimal allocation, including cross-modal misallocation between textual and visual evidence and intra-visual imbalance among visual tokens. We propose RAVE (Re-Allocating Visual Attention), a lightweight pair-gating mechanism that adds a learned query–key bias to pre-softmax attention scores over visual keys, derived from pre-RoPE query and key features. RAVE requires no architectural modification to the backbone and can be trained end-to-end with the rest of the model. Across a suite of multimodal benchmarks, RAVE improves over standard attention by 3 points on average, with the largest gains on perception-intensive tasks—including multilingual OCR, chart understanding, document VQA, and scene text VQA—where accurate visual grounding is critical.

More from EMNLP 2026

All EMNLP 2026 papers →