A Study of Real-world Audio-Visual Corpus Design and Production: A Perspective from MISP Challenges
Abstract
With the increasing proliferation of speech-driven applications, the challenges in their deployment environments are becoming more prominent. Accordingly, audio-visual speech processing (AVSP) proposed integrating audio and visual information to enhance performance across many speech-processing tasks. In this context, the MISP challenges were organized at ICASSP 2022, 2023, and 2024, respectively. These challenges released audio-visual corpora to support four core tasks: audio-visual wakeup, diarization, speech enhancement, and recognition. The datasets have garnered attention from the global research community, with over 110 teams downloading the corpora. This paper provides a comprehensive analysis of the MISP corpus design from various perspectives, including scenario selection, recording equipment and processes, as well as manual transcription and alignment, highlighting its strengths and limitations and offering insights and recommendations for the design of future AVSP corpora.