MeRP: Efficient Multi-Expert Routing Prompt Encoding for Multilingual Translation with LLMs
Abstract
AbstractLarge Language Models (LLMs) enable a single model to support translation across multiple language directions simultaneously, eliminating the need to train a separate model for each direction and substantially improving the performance of multilingual machine translation. However, during multilingual translation instruction tuning, parameter interference among different languages significantly limits the improvement of translation performance and may even cause performance degradation for some languages. To address this issue, we propose MeRP, a multi-expert routing prompt encoding strategy for parameter-efficient fine-tuning (PEFT) in multilingual machine translation tasks. This method encodes representations of source languages through an attention mechanism and dynamically selects expert groups according to translation directions, enabling both the sharing and disentanglement of multilingual semantic representations. As a result, it preserves LLM generalization while reducing training time and improving translation performance. We conduct experiments on five language pairs. The results show that, compared with the baseline models, our method updates only 0.06% of the total model parameters during training while achieving better multilingual translation performance. Notably, our MeRP reduces training time by 46.75% and achieves average gains of +0.62 BLEU points and +0.34 COMET-22 points over the arithmetic mean of the four PEFT baselines.Code is available at https://github.com/wangzhenhan93/MeRP.