Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
Abstract
AbstractLarge language models (LLMs) are increasingly used for automatic evaluation of generated text, but most work focuses on English. Extending LLM-based evaluation to multilingual settings is challenging, especially for low-resource languages or when in-domain data is scarce. This work explores how an English evaluation corpus can serve to evaluate other languages, both when in-domain data is available for fine-tuning and when it is not. We systematically analyze English, Spanish, and Basque, representing high-, mid-, and low-resource languages, considering instruction translation, monolingual versus multilingual supervision, and model size. Our results reveal key trade-offs: When in-domain data is available, fine-tuned smaller models can achieve performance comparable to proprietary models, whereas zero-shot evaluation with larger models proves more effective in out-of-domain settings. We also observe that fine-tuning on out-of-domain data can adversely affect model performance. These findings provide practical guidance for building efficient, reliable multilingual evaluation pipelines. The data and code are publicly available at hitz-zentroa/mJudge .