Skip to content
conftrace_
2026 EMNLP EMNLP 2026

How does Multilingual Finetuning of Instruction-Tuned Large Language Models Affect Languages in Layer-Wise Hidden Representations?

AbstractContemporary large language models (LLMs) exhibit multilingual capabilities but remain predominantly English-centric, resulting in a persistent performance gap across languages. While some approaches to multilingual training are grounded in the implicit pivot language hypothesis to improve interpretability, it remains underexplored how the languages represented in each layer are affected during training, particularly in practical, cost-efficient instruction-tuning scenarios. In this study, we investigate how instruction tuning with Low-Rank Adaptation (LoRA) in a target language affects the layer-wise language representation of instruction-tuned LLMs, with the aim of assessing the relevance of the implicit pivot language hypothesis as a theoretical grounding for multilingual finetuning. We find that target-language instruction tuning shifts models’ internal language representation in a non-uniform manner across layers and sublayers: middle layers shift further toward English, while early and final layers primarily serve the mapping between the target language and the dominant intermediate language. Our findings empirically support the implicit pivot language hypothesis in the context of instruction tuning, demonstrating that layer-wise parameter variation broadly corresponds to the hypothesized functional roles of layers in multilingual processing. Our code is available at: https://github.com/hyemijo/layer_interpretability

❓ The Questioner
🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →