Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders
Abstract
AbstractMultilingual Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear. Do they form shared multilingual representations with language-specific decoding, and if so, why do multilingual performance gaps persist? To address this question, we train models on different multilingual mixtures and analyze their internal mechanisms using Cross-Layer Transcoders (CLTs) and Attribution Graphs. Our results reveal a consistent three-phase structure across models: early layers encode language-specific information, middle layers converge to highly shared multilingual representations, and final layers re-specialize for language-specific decoding.Training models without English produces the same shared multilingual structures, suggesting that these representations emerge independently of English dominance. We further show that decoding relies on a small set of high-frequency late-layer features that linearly encode language identity from earlier layers, and that intervening on these features allows one language to be suppressed and another substituted. Finally, through a Model-Diffing experiment, we show that non-English failures arise from weak late-layer decoding features, weak middle-layer clusters, and tokenizer fragmentation that forces early layers to specialize in word reassembly. Finetuning strengthens these features and their connections, improving token assembly and language-specific decoding, providing a mechanistic explanation for multilingual performance gaps.