Inside Sentence Embeddings: Leveraging Token Representations for Machine Translation Evaluation and Word Alignment
Abstract
AbstractContextualized token-level embeddings from encoders, e.g., mBERT, are used for several cross-lingual tasks, e.g., machine translation (MT) evaluation and alignment tasks, but fine-tuning is required for the specific task settings. In contrast, multilingual sentence embeddings are trained under sentence-level objectives on parallel data, but their token-level representations remain unclear whether they carry cross-lingual semantics, although not explicitly trained with token-level objectives. In this paper, we investigate token representations extracted from multilingual sentence embedding models for MT evaluation and word alignment. Experimental results and analysis show that sentence embedding models can provide useful token-level representations for direct similarity-based tasks. Although the observed advantage is not consistent across all settings, the results indicate that multilingual sentence-level training may support token-level semantic similarity without explicit token-level training.