Cross-lingual Representation Learning via Centroid Intervention Fusion
Abstract
AbstractLarge language models (LLMs) exhibit uneven multilingual performance, especially when dealing with low-resource languages. Inference-time intervention offers a lightweight way to improve cross-lingual transfer by modifying the hidden states produced by the LLMs during the forward pass, without updating model parameters. However, existing cross-lingual intervention methods typically learn separate projections from source to target languages, which limits scalability and prevents knowledge sharing across languages. We propose Centroid Intervention Fusion (CIF), a projection fusion framework that consolidates multiple multilingual intervention projections into a single language-shared operator. CIF builds a stable language-shared intervention operator by estimating a robust centroid over multilingual projections and filtering outliers. Across multilingual commonsense reasoning, natural language inference, factual editing, and machine translation benchmarks, CIF outperforms the strongest prior pairwise intervention baseline by up to +3.378 pp on average across four model backbones. By preserving shared cross-lingual intervention representation, CIF further improves performance on low-resource languages. The code will be released upon acceptance.