SA-RAS: Speaker-Aware Style Retrieval Augmented Generation for Expressive Zero-Shot Text-to-Speech Synthesis

Zero-shot text-to-speech (TTS) models can clone an unseen speaker by reference speech. Given multiple reference speeches, Retrieval Augmented Generation (RAG) shows potential in enhancing synthesis quality. The existing method faces two key limitations: semantically-biased retrieval tends to introduce style mismatch, and the deterministic mapping between text and speech fails to fully capture the unique style of the speaker. To solve these, we propose a Speaker-Aware Retrieval Augmented TTS Synthesis (SA-RAS) model. It consists of a two-stage TTS model, Split-TTS and a zero-shot CLAP. The former embeds style and timbre separately to obtain a style space for retrieval. The latter optimizes the correlation between text and style by incorporating speaker information into the retrieval model. Experiments prove that SA-RAS boosts the style expressiveness of synthesized speech.