Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Learning to Reason Before Masking for Referring Audio-Visual Segmentation

AbstractExisting referring Audio-Visual Segmentation (Ref-AVS) benchmarks primarily focus on explicit expressions whose referents can be resolved through direct visual, acoustic, or linguistic correspondence. Consequently, they provide limited coverage of implicit references that require multi-step reasoning over multimodal evidence. To bridge this gap, we introduce Reasoning Ref-AVS, a new task that requires models to infer the referred object from audio-visual-language cues before generating its segmentation mask. We further construct ReAVS, a comprehensive dataset with manually annotated referring expressions, multimodal cues, object masks, and reasoning chains, covering diverse reasoning spanning diverse reasoning patterns. Evaluations of representative Ref-AVS models on ReAVS reveal substantial performance degradation when direct cross-modal correspondence is insufficient, exposing a clear gap between multimodal recognition and referent reasoning. Motivated by these challenges, we propose Re2Seg, an omni-modal framework that performs object-centric evidence encoding, multi-hop reasoning, and reasoning-conditioned mask decoding. Experiments demonstrate that Re2Seg achieves stronger segmentation performance under reasoning-intensive references and provides interpretable object-inference traces. These results highlight the importance of learning to reason before masking for bridging the gap between multimodal recognition and referring segmentation.

More from EMNLP 2026

All EMNLP 2026 papers →