Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Domain-Based Routing in Mixture-of-Experts Models for BabyLMs

AbstractRecent language model improvements were mostly achieved by increasing model scale, compute, and in particular training data size. This differs significantly from human language acquisition, which relies on significantly fewer words. The BabyLM challenge tackles this issue by encouraging approaches inspired by cognitive insights to reduce data dependency, thus lowering the barrier for NLP researchers to contribute to their field. We propose a Mixture-of-Experts approach with guided domain-based routing to learn subnetworks specialized on human-interpretable topics, thus making experts focus on domain knowledge and style rather than specific token features. Additionally, we apply a CEFR-based curriculum learning to mimic human language learning. Our model underperforms compared to the 7 times larger baseline model, but scores best on the BabyLM strict-small leaderboard among models with less than 25M parameters. Our code is publicly available at https://github.com/SmallLM-DBR-MoE-Submission-2026/Baby-DBR, and our model can be found at https://huggingface.co/elschr/BabyLM2026-DBR.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Interdisciplinary and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →