Skip to content
conftrace_
2026 EMNLP EMNLP 2026

M3 Scaling Law: Optimizing Multi-Epoch, Multi-Lingual, and Multi-Stage Training for Low-Resource Language Models

AbstractIn this paper, we study a fundamental design problem in pretraining Large Language Models (LLMs) for low-resource language regimes, focusing on cross-family language pairs in which English serves as the high-resource language. Existing works adopt multi-epoch, multi-lingual, and multi-stage training to utilize the limited target-language corpus efficiently, but no prior scaling law can compare recipes spanning these approaches under the same compute budget C and target-language corpus size DT, leaving the optimal training setup unclear. To address this gap, we propose the M3 Scaling Law, a unified predictive model parameterized by the model scale, the number of target-corpus epochs k, the average target-language ratio r, and the final-stage target-language ratio rf, which places monolingual single-stage, multi-lingual single-stage, and multi-lingual multi-stage recipes on a single target-language validation-loss surface. Across three language pairs (English paired with Japanese, Indonesian, and Swahili), it extrapolates to unseen hyperparameter regions more accurately than existing scaling laws. Using M3 as a surrogate objective, we derive two practical guidelines for low-resource LLM pretraining: (i) as DT decreases, the optimal recipe shifts directly from monolingual single-stage to multi-lingual two-stage training at a compute-budget-dependent threshold, with multi-lingual single-stage never optimal in our experimental grid; and (ii) the optimal number of epochs collapses onto a single curve in the scarcity variable DT/D\*(C), where D\*(C) ∝ C𝛼/(𝛼+𝛽) is the monolingual compute-optimal corpus size.

More from EMNLP 2026

All EMNLP 2026 papers →