Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Architecture-Conditioned Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning

AbstractThe rapid progress of reasoning and agentic large language models (LLMs) has increased the demand for long-context inference, but self-attention (SA) scales quadratically with context length. To address this, we propose SWARR (Sliding-Window Attention with Reinforced Adaptation for Math Reasoning), a practical recipe for adapting SWA models to mathematical reasoning. SWARR has two stages: (1) efficient conversion from a pretrained SA model to SWA with supervised fine-tuning (SFT), which avoids pretraining a new base model, and (2) architecture-aware policy adaptation with reinforcement learning (RL). We find that SWA still underperforms SA after SFT, and we hypothesize that this gap is caused in part by a data-architecture mismatch: most SFT data are prepared for SA models and may contain long-range dependencies that are difficult for SWA to model. Because RL is architecture-aware and optimizes self-generated trajectories under the SWA constraint, it can adapt training trajectories to better match the SWA architecture. Experiments on mathematical reasoning benchmarks show that this architecture-aware RL substantially narrows the gap between SWA and SA, recovering much of the accuracy lost during SWA conversion while preserving the efficiency benefits of linear-complexity attention.

More from EMNLP 2026

All EMNLP 2026 papers →