FECE: A Challenge Set for Evaluating Fine-Grained Translation Error Correction
Abstract
AbstractExisting machine translation evaluation metrics assess translation quality only at the segment level and therefore cannot assess fine-grained translation quality. For translation error correction systems, this means that the quality of fine-grained corrections cannot be evaluated using existing metrics. To highlight this evaluation challenge, we propose the FECE dataset for evaluating fine-grained correction quality. Experiments on FECE show that existing metrics do not always reflect fine-grained correction performance. To enable the evaluation of fine-grained correction performance, we further propose COMET_Fine, a fine-grained evaluation metric. COMET_Fine can effectively assess fine-grained correction quality while also revealing inconsistencies between existing metrics and COMET_Fine.