Towards Efficiently Whisper Fine-tuning with Monotonic Alignments

Whisper is a popular large-scale model developed by OpenAI, which can perform robust speech recognition and speech translation tasks. Fine-tuning Whisper on various domains thus becomes a common approach to improve the model performance on specific tasks. This paper introduces an efficient fine-tuning method that enhances Whisper's fine-tuning quality by incorporating a training objective based on monotonic alignments, without modifying the model network. Experimental results show that the proposed fine-tuning approach achieves consistent and considerable improvements over normal fine-tuned models on speech recognition tasks and some of the speech translation tasks. In addition, a new dataset MDCCST is built to validate the performance of Cantonese-to-Mandarin translation.