Skip to content
conftrace_
2026 EMNLP EMNLP 2026

An Analysis of Source-Side Diversity in Machine Translation Evaluation

AbstractBenchmarks are central to assessing progress in machine translation (MT), yet the content of widely used MT test sets is increasingly scrutinized. Concerns about data contamination and benchmark difficulty have gained traction, but the role of source-side diversity remains largely overlooked. Many popular benchmarks contain overlapping source items and even exact duplicates, raising the question of how such repetition affects evaluation quality. We empirically examine how source-side diversity shapes MT evaluation. First we measure the inter-source diversity of nine popular MT datasets. Next, our downstream analysis, based on WMT24, suggests that benchmarks with low diversity may be less discriminative and may provide artificially narrow confidence intervals. Ultimately, we call for greater attention to inter-source diversity in general MT benchmark design.

More from EMNLP 2026

All EMNLP 2026 papers →