2026
EMNLP
EMNLP 2026
Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling
Abstract
AbstractNeural language model architectures that iterate over their layers have recently shown superior performance on reasoning benchmarks and proved to be a promising method for reducing model size while maintaining performance at scale. However, the effectiveness of such architectures in data-constrained settings remains unexplored. We present a systematic study of layer-sharing transformer architectures trained on the BabyLM corpus (10M tokens), across various iteration strategies and a range of layer and iteration counts. We find that iterative architectures are highly effective under data-constrained setting: a 3-layer iterated model outperforms a standard 24-layer transformer model (ModernBERT-inspired) on SuperGLUE benchmark.