Temporally Streaming Audio-Visual Synchronization for Real-World Videos

We introduce RealSync a novel dataset designed to significantly enhance the training and evaluation of models for audio-visual synchronization (AV Sync) tasks. Sourced from high-quality YouTube channels RealSync covers a wide range of content domains providing an improved scale diversity and alignment with broadcast content compared to existing datasets. It features extended-length video samples catering to the critical need for more comprehensive real-world training and evaluation materials. Alongside this dataset we present StreamSync a model tailored for real-world AV Sync applications. StreamSync is designed to be backbone agnostic and incorporates a streaming mechanism that processes consecutive video segments dynamically iteratively refining synchronization predictions. This innovative approach enables StreamSync to outperform existing models offering superior synchronization accuracy with minimal computational cost per iteration. Together our dataset and the StreamSync model establish a new benchmark for AVSync research promising to drive the development of more robust and practical AVSync methods. https://github.com/jvoas655/StreamSync