Skip to content
conftrace_
2026 EMNLP EMNLP 2026

SUNDER: Selective Unmasking for Text Understanding

AbstractDecoder-only large language models (LLMs) employ autoregressive text processing, attending exclusively to preceding tokens when generating subsequent ones. Although this architectural design excels in text generation applications, it constrains model performance on certain text understanding tasks. In this work, we propose SUNDER, a novel approach that enables partial bidirectional attention computation. We validate the proposed approach on three token spans identification tasks, such as Named Entity Recognition (NER), Aspect-based Sentiment Analysis (ABSA), and Definition Extraction (DEF). Results show that SUNDER improves LLM performance by up to 5 percentage points compared to traditional instruction-tuning. We further validate the robustness of our approach on multiple model families, model sizes, and prompting techniques. Our evaluation extends to multilingual scenarios and single-hop question answering (QA), confirming SUNDER’s effectiveness on multiple settings. The code is made available to the community.

More from EMNLP 2026

All EMNLP 2026 papers →