Reasoning Beyond English: Separating Capability and Language in Cross-Lingual Reasoning Model Adaptation
Abstract
AbstractMany top-performing reasoning language models (RLMs) exhibit English-centric reasoning behavior, generating English chains of thought even when the prompt and final answer are in another language. Beyond its impact on multilingual reasoning performance, this also blocks downstream uses of the reasoning trace, such as debugging model errors and auditing decisions. We studied different approaches to RLM language adaptation by adapting Qwen3-4B-Thinking-2507 into Polish. We find that the accuracy collapse of SFT decomposes additively into termination and quality failure modes. On the other hand, KL-anchored fine-tuning mitigates these failures but exhibits a sharp plateau-then-cliff structure under reverse divergence: language match collapses from 0.89 to 0.37 as 𝛼 doubles, with no comparable structure under forward divergence. However, a second training stage leveraging GRPO helps close the remaining gap and recover the language match even from configurations past the cliff, revealing that the two stages divide labor: the supervised stage controls reasoning quality while the RL stage closes residual language match gaps. These findings suggest that, in a two-stage RLM post-training, the supervised phase can be considerably less constrained than current practice assumes.