Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets

AbstractWe release **morphoBPE**, a framework for training and comparing per-language BPE tokenizers across 242 Latin- and Cyrillic-script languages built from Wikipedia *glottosets* (per-language TF-DF lexicons), with four diagnostic evaluations of what BPE captures across scripts. Unsupervised language identification by token count reaches 13.8% accuracy across 320 Latin-script languages, against 19.83% for supervised fastText ‘lid.176‘, which returns zero for 147 of them because they fall outside its 176-language alphabet, including our best performers (Lak 81.5%, Cree 80.6%, Inuktitut 63.8%, Kabardian 60.1%). BPE aligns with morpheme boundaries at F1 = 0.28 across 15 languages (Morfessor: 0.43), and beats supervised character-*n*-gram cross-lingual transfer by +37% F1 across six pairs where labeled morphology is unavailable. BPE distance correlates moderately with Glottolog phylogeny (Mantel r = 0.329). We position per-language BPE as a coverage-first diagnostic for languages outside the supervised long tail, not as a substitute for supervised methods where labels exist.

More from EMNLP 2026

All EMNLP 2026 papers →