Skip to content
conftrace_
2026 EMNLP EMNLP 2026

What Does Layer-Importance Reveal About Transformers and State-Space Models?

AbstractTransformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuning, and interpretability across both families. We decompose layer importance into two distinct notions. Necessity captures how much the pretrained model depends on a layer’s existing contribution, measured by the loss increase from bypassing it. Plasticity captures where the model absorbs new information during fine-tuning, measured by the magnitude of task-specific weight updates. Our analysis reveals that transformers and SSMs behave fundamentally differently: in residual transformers, Necessity and Plasticity anti-align across depth, whereas in Mamba-style SSMs they point to overlapping regions. The sign of this alignment also predicts downstream adaptation behavior. In transformers, the updates are in the most plastic layers, which increases catastrophic forgetting, while this tier-dependent effect disappears in Mamba-style SSMs

More from EMNLP 2026

All EMNLP 2026 papers →