Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Tokens of Meaning: Semantically-Informed Morphological Tokenization in Low-Resource Language Modeling

AbstractSubword tokenizers for language models are typically trained to efficiently encode text into short sequences of reusable units. These often produce tokens which are not linguistically meaningful, such as giraffe –> g ira ffe. We ask whether a tokenizer that instead splits words into semantically meaningful morphemes can help a language model learn under a developmentally plausible data budget, where every training token is scarce and sharing evidence across related word forms should matter most. We build such a tokenizer for English by hand-curating a word-to-morpheme-segmentation dictionary, built on top of MorphyNet, a database of derivational and inflectional morphology. Each word is only split into morphemes when the pieces are productive and the whole word’s meaning is semantically related to them, thus splitting unhappiness into un happy ness, but keeping business whole. On the 10M-word BabyLM corpus this holds out-of-vocabulary tokens to 1.0%, with an inventory of ~37k tokens. We compare the tokenizer intrinsically against both statistical (BPE, Unigram, WordPiece) and morphology-aware (Morfessor, MorphPiece) baselines, and measure its effect on a child-scale language model. In a controlled comparison against an uncurated baseline built from the same morphological source, curation yields +1.4 BLiMP. Our submitted model, which adds masked next-token prediction and vocabulary pruning, reaches 72.53 BLiMP and ranks second on COMPS and third on EWoK on the BabyLM 2026 strict-small leaderboard.

🌉 Interdisciplinary Bridge - Interdisciplinary and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →