J-j-j-just Stutter: Benchmarking Whisper's Performance Disparities on Different Stuttering Patterns

Despite their prevalence in everyday technologies, automated speech recognition (ASR) systems often struggle with disfluent speech. To diagnose and address these technical challenges, we evaluate OpenAI's Whisper, a state-of-the-art ASR model, using speech samples from podcasts with people who stutter. Our results show significant disparities in Whisper's performance between fluent and stuttered speech. Within disfluent speech, Whisper performs significantly worse on speech with sound repetitions - a disfluency more unique to stuttering. Notably, sound repetitions not only lead to transcription mistakes but also trigger Whisper to hallucinate over 20% of the time. Conducted by researchers who stutter, this study brings new insights on ASR biases against disfluent speech and highlights the value of disability-led research in addressing technological inequities affecting people with disabilities.