Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Beyond Surface Form: Probing Script and Discourse-Marker Sensitivity in Marathi LLM Reasoning

AbstractEvaluation of multilingual large language models has expanded rapidly, yet aggregate benchmark scores often conceal how performance shifts once the surface form of a language changes. We examine this question through Marathi discourse relation classification. Five instruction-following models, Llama-3.1-8B, Llama-3.3-70B, Qwen3.6-27B, GPT-OSS-20B, and GPT-OSS-120B, are evaluated on a manually curated Marathi language dataset spanning six discourse relations, two scripts, eighteen domains, and three difficulty levels, producing 2500 model-level predictions. Qwen3.6-27B obtains the highest overall accuracy at 82.4%, followed by GPT-OSS120B at 80.8%. The central finding is a large script effect as mean accuracy falls from 78.1% on Devanagari script to 49.7% on Romanized Marathi, a decline of 28.5 percentage for content that is otherwise unchanged. Relationlevel results trace to marker form rather than presence; every instance carries an explicit marker, but Temporal and Contrastive relations use a single canonical connective and are handled reliably, while Inferential and Reformulative relations lean on multi-word phrasal markers and are frequently confused with Causal, a 20.4-point accuracy gap that holds across all five models. Llama-3.1-8B shows a more severe failure, defaulting to Contrastive across most gold relations. These results indicate that strong multilingual performance on Marathi is neither script-invariant nor uniform across discourse relations.

More from EMNLP 2026

All EMNLP 2026 papers →