Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Does Item Response Theory Generalize Across Languages? Evidence from Cross-Lingual LLM Benchmarks

AbstractThe application of Item Response Theory (IRT) for data selection in benchmarks has mostly been limited to monolingual settings. In this work, we ask: how does IRT generalize when the same benchmark is translated across languages? We conduct experiments on Multilingual GSM and Global MMLU, covering over 100 language models and 11 languages. Fitting 1PL, 2PL, and 3PL IRT models per language, we assess cross-lingual consistency of parameters and whether IRT-filtered subsets preserve model rankings across languages: item *difficulty* emerges as the most stable parameter followed by *guessing* and *discrimination*. A differential item analysis also shows that lack of stability can be tracked to specific items shifting under translation across languages. Our findings suggest IRT-based selection in multilingual settings must be both parameter-aware and item-aware, validating model choice and screening individual items before assuming cross-lingual generalization.

More from EMNLP 2026

All EMNLP 2026 papers →