Text4Speech: Text-Driven Training for Massively Multilingual Speech Understanding
Abstract
AbstractSpeech is a primary communication interface for humans, and thus a crucial modality for large language models (LLMs). Despite this, the vast majority of open speech-capable LLMs are developed for and evaluated on a small set of <20 high-resource languages, adding to the already massive inequity of modern language technology. Massively multilingual speech understanding is primarily hindered by the limited amounts of speech training data for most languages and tasks. For the lion’s share of the world’s languages, speech datasets for tasks other than automatic speech recognition (ASR) simply do not exist. In this work, we bypass the sparsity of task-specific speech data by leveraging text – much more available for many languages – to enable an LLM to understand speech input. Our approach, dubbed Text4Speech, leverages state-of-the-art multilingual pre-aligned text and speech encoders: we first align the multilingual text encoder to an LLM using abundant text data and then further align the speech encoder to the LLM using a limited amount of speech. In both steps, we train the alignment parameters via self-distillation on the LLM. Our evaluation on three multilingual speech understanding benchmarks that encompass 90 languages renders Text4Speech highly effective, yielding average improvements of up to 6% on Fleurs-SIB over a very strong ASR-based cascade baseline. Crucially, we show that our text-based alignment unlocks a highly sample-efficient alignment of the speech encoder to the LLM.