Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Can Randomly Permuting the LM Decoding Order Boost Data-Efficiency?

AbstractCausal language models, typically trained with next-token prediction, condition each token’s probability on all preceding tokens. When pretraining is limited by the amount of available training data (rather than compute), repeating training over the same corpus yields a largely unchanged learning signal across epochs. In contrast, permutation language models (PLMs) predict tokens in a random order, conditioning each prediction on a bidirectional set of tokens and thereby increasing the diversity of contextual evidence a token can receive. We hypothesize that PLMs (1) profit more from longer training on repeated data, (2) outperform classical CLMs in data-limited settings where multiple epochs are allowed, and (3) excel on tasks where bidirectional context is useful.To test these hypotheses, we train multiple models on the BabyLM strict-small pretraining data consisting of only 10M words and evaluate how decoding order and bidirectional conditioning affect model performance under tight training-data budgets. We find that while PLMs profit from longer training, they do not generally outperform CLMs except on tasks where use of bidirectional context is required.

More from EMNLP 2026

All EMNLP 2026 papers →