Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling

AbstractNeural language model architectures that iterate over their layers have recently shown superior performance on reasoning benchmarks and proved to be a promising method for reducing model size while maintaining performance at scale. However, the effectiveness of such architectures in data-constrained settings remains unexplored. We present a systematic study of layer-sharing transformer architectures trained on the BabyLM corpus (10M tokens), across various iteration strategies and a range of layer and iteration counts. We find that iterative architectures are highly effective under data-constrained setting: a 3-layer iterated model outperforms a standard 24-layer transformer model (ModernBERT-inspired) on SuperGLUE benchmark.

More from EMNLP 2026

All EMNLP 2026 papers →