Are Multilingual Models Improving? Revisiting Cross-Lingual Transfer
Abstract
AbstractCross-lingual transfer is a model’s ability to generalize capabilities from well-represented source languages to under-represented target languages. Existing measures to quantify a model’s transfer strength conflate improvements in transfer with general improvements to accuracy in source language. We advocate an alternate metric that reliably captures transfer strength called Hardness Adjusted Transfer (HAT) Score, and use it to derive multiple insights on factors influencing transfer strength. Our analysis with twenty diverse language models and three popular mainstream multilingual benchmarks argues that 1) transfer in small models is not broken, 2) we are making slower than expected progress with model size, and 3) have made clear progress over time.