Skip to content
conftrace_
2026 EMNLP EMNLP 2026

MUA: Multiple Context Units Attention For Large Language Models

AbstractLarge language models (LLMs) are increasingly required to process inputs composed of multiple context units, such as answer options, retrieved passages, or in-context demonstrations. Effective reasoning in such scenarios demands capturing both intrinsic semantics within each unit and the aggregated semantics across units. However, recent efforts typically treat multiple units as a single entangled sequence, ignoring the distinct knowledge within isolated units, which leads to suboptimal performance. To this end, we propose MUA(Multiple Context Units Attention), a dual-view self-attention module that computes a local view for learning unit-specific semantics and a global view to enable bidirectional interaction across units. These two views share projection matrices and are fused by a lightweight input-conditioned gate, while generation proceeds with standard causal decoding. With an optimized implementation, this design achieves efficient fusion of multiple context units and superior information utilization during input encoding at marginal overhead. Experiments on diverse multiple context units reasoning benchmarks, including multiple-choice, multi-hop QA and language understanding tasks, show that MUA consistently improves performance across model scales, exhibits increased robustness to context units reordering, and establishes new state-of-the-art results on key benchmarks.

More from EMNLP 2026

All EMNLP 2026 papers →