Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Cross-Lingual Activation Steering for Visual-Prior Conflicts

AbstractMultimodal large language models (MLLMs) must often arbitrate between visual evidence and world-knowledge priors when an image conflicts with commonsense expectations. Existing work has demonstrated that activation steering vectors can bias model outputs toward either source of information, but it remains unclear whether this arbitration mechanism generalizes across languages. To investigate this, we construct a multilingual extension of the Visual CounterFact color split and derive activation steering directions that contrast image-grounded answers with prior-based answers. Across 12 languages, we find that English-derived and shared multilingual steering directions consistently shift model outputs on non-English prompts, suggesting that visual-prior arbitration has a substantial cross-lingual component. At the same time, we find that performance gains on common knowledge questions can come at the cost of image-grounded accuracy, a trade-off that is particularly pronounced for lower-resource languages. Code and data are available at https://github.com/elffiona/cross-lingual-visual-prior-steering.

More from EMNLP 2026

All EMNLP 2026 papers →