Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Spend the Wasted Epochs: Preference Post-Training from the Saturation Checkpoint for Sample-Efficient Language Modeling

AbstractSample-efficient pretraining under a developmental data budget, such as the ten million words and ten epochs of the BabyLM Strict-Small track, is usually approached by adding training signal: a new objective, optimizer, curriculum, or teacher. The budget itself receives less scrutiny. We show that much of it is spent after learning has stopped. A 33.0M-parameter GPT-BERT hybrid trained under the track rules saturates on downstream evaluations by roughly 70M words seen, about 70% of its permitted exposure, while training loss continues to fall; the final epochs improve fit, not competence. From this diagnosis we derive three design principles, timing, preservation, and alignment, and instantiate them as Minimal-Pair Preference Optimization (MPO): a short post-training phase, begun at the saturation checkpoint, that teaches the model to prefer real corpus sentences over length-preserving corruptions of them under a frozen-reference penalty. A controlled study of seven in-training interventions supports the principles: none improves on the baseline, and the unanchored ones damage the same cluster of long-distance dependencies first. MPO improves five of the seven official task averages, concedes under a point on the other two, is alone in repairing that cluster, and matches the fully trained baseline with 7.2 rather than 10 epochs.

More from EMNLP 2026

All EMNLP 2026 papers →