Skip to content
conftrace_
2026 EMNLP EMNLP 2026

What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling?

AbstractLearned weighting of earlier layers, as used in ELC-BERT (Charpentier and Samuel, 2023), is a promising direction for sample-efficient language modeling. Attention Residuals (AttnRes; Kimi Team et al., 2026) extend this idea by computing input-dependent weights separately for each token, with reported gains in large-scale autoregressive models. We present a controlled study of whether such input-dependent depth routing helps in the data-limited regime where static routing first proved useful. Across the 10M- and 100M-word BabyLM budgets and three paired seeds, AttnRes consistently lowers test negative log-likelihood (NLL) compared to a residual baseline. It reaches the same development-set (dev) NLL after seeing fewer training words. In the 10M L32 control, AttnRes also lowers test NLL relative to static routing in all three seeds. Behavioral differences between AttnRes and the baseline are small and mixed. The BLiMP (Warstadt et al., 2020) advantage at equal training exposure disappears after matching on dev NLL. The trained 10M-word AttnRes models make causal use of routes from nonrecent depth blocks, but we find no clear evidence that categories with greater dependence on these sources achieve larger BLiMP gains over the baseline.

More from EMNLP 2026

All EMNLP 2026 papers →