Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Small Models, Fewer Heads, More Forgetting: Sparse Attention Fine-Tuning for SLM Continual Learning

AbstractSmall Language Models (SLMs) have been widely deployed in edge computing and mobile devices. Generally, these models need to be continuously updated to adapt to dynamic environments and evolving user needs. Therefore, the performance of SLMs in continual learning has become a critical research focus. Compared to LLMs with rich redundant representation spaces, SLMs are more likely to overwrite previous representations during new task learning due to their fewer attention heads and layers. To address this issue, we propose LightLoRA, a sparse fine-tuning method specifically designed for SLMs. This method constructs a conflict-aware importance mechanism by combining raw importance from SVD decomposition with task conflict degrees based on gradient direction consistency. By selecting and updating heads that provide high contributions and low conflicts, LightLoRA successfully maintains sufficient space for knowledge preservation. Experimental results show that the proposed method reaches state-of-the-art performance across multiple benchmarks without adding extra parameters. Code is released on https://anonymous.4open.science/r/LightLoRA-077C.

More from EMNLP 2026

All EMNLP 2026 papers →