Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Script Matters: Benchmarking Open-Source Machine Translation for Manipuri in Meetei-Mayek and Bengali Script

AbstractManipuri (Meiteilon) officially recognised the Meetei-Mayek script alongside the Bengali script in 2021, with both scripts permitted for concurrent use during a 10-year transition period. However, the FLORES-200 devtest release still encodes Manipuri only as Bengali script, leaving the language’s current official script without a standard benchmark. We address this gap by contributing the Meetei-Mayek script (mni_Mtei) layer, aligned with the existing Bengali-script (mni_Beng) data, to form a four-way parallel resource comprising English, Hindi, Meetei-Mayek Manipuri, and Bengali-script Manipuri, with 1,012 sentences per language. This resource enables separate evaluation of open-source MT and LLM systems on the two Manipuri scripts, rather than treating them as a single "Manipuri" target. Across three IndicTrans2 checkpoint families, Sarvam-Translate, and five general-purpose LLMs, evaluated using six automatic metrics, we find that legacy Bengali-script output often scores higher than Meetei-Mayek despite the latter being the official script; IndicTrans2’s family degrades sharply on Bengali-script Manipuri; Sarvam-Translate cannot produce Bengali script at all; and none of the five general-purpose LLMs we test can reliably read or write Meetei-Mayek. Script identity is therefore a first-class evaluation variable for Manipuri MT: a single "Manipuri" score can mask large, direction-dependent quality gaps.

More from EMNLP 2026

All EMNLP 2026 papers →