Online AV-CrossNet: a Causal and Efficient Audiovisual System for Speech Enhancement and Target Speaker Extraction

This paper presents online AV-CrossNet, a computationally efficient audiovisual speech enhancement/extraction system capable of causal and real-time processing. We aim to improve the state-of-the-art AV-CrossNet by enabling causal, frame-by-frame processing. To achieve this, we incorporate causal layers and compression techniques, reduce model size, and employ only one-frame look-ahead, thereby substantially enhancing real-world applicability. Additionally, we analyze compression ratio in both audio and visual modules, providing valuable insights into audiovisual model compression. Experimental results demonstrate an inference latency of 4.73 ms, capable of real-time processing. Moreover, the system maintains competitive performance while reducing size by a factor of 10. These findings highlight the efficiency and effectiveness of the proposed system, offering a promising solution for real-time audiovisual speech enhancement and speaker extraction in acoustically adverse environments.