Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Beyond Aggregate Loss: Training Dynamics and Language-Specific Scaling Laws for Multilingual Language Models

AbstractScaling laws typically characterize the performance of large language models (LLMs) through aggregate loss, potentially obscuring substantial variation across languages in multilingual pretraining. We study this variation through controlled experiments on a diverse set of languages spanning two language families, training 0.3B- and 0.5B-parameter models on up to 137B tokens. Specifically, we compare continued pretraining from a strong pretrained model with training the same architecture from scratch, examine the effects of expanding the multilingual corpus, analyze cross-family transfer through language-specific loss trajectories, and fit separate scaling laws for individual languages. Our results show that continued pretraining from a strong pretrained model leads to faster convergence and consistently stronger downstream performance than training from scratch. In contrast, reductions in training loss and more stable gradient ℓ2 norms do not necessarily translate into downstream benchmark gains. We further demonstrate that training on a single language family can reduce loss for languages in another family, providing evidence of cross-family transfer. Finally, we find that data-scaling behavior is relatively consistent across languages, whereas the benefits of model scaling vary substantially. A sampling-weighted combination of the language-specific scaling laws closely reconstructs the aggregate training loss. These findings show that aggregate metrics provide an incomplete account of multilingual pretraining dynamics and highlight the importance of language-specific analyses for evaluating and scaling multilingual models.

More from EMNLP 2026

All EMNLP 2026 papers →