Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations
This addresses the problem of speech enhancement for real-time applications like video conferencing, though it is incremental by building on existing AVSR and ASD methods.
The paper tackled speech enhancement in noisy, multi-speaker environments by developing RAVEN, a real-time audio-visual system that isolates and enhances a target speaker using pre-trained visual embeddings, showing improvements in low-SNR conditions with multiple interfering speakers and noise-only scenarios.
Speech enhancement in audio-only settings remains challenging, particularly in the presence of interfering speakers. This paper presents a simple yet effective real-time audio-visual speech enhancement (AVSE) system, RAVEN, which isolates and enhances the on-screen target speaker while suppressing interfering speakers and background noise. We investigate how visual embeddings learned from audio-visual speech recognition (AVSR) and active speaker detection (ASD) contribute to AVSE across different SNR conditions and numbers of interfering speakers. Our results show concatenating embeddings from AVSR and ASD models provides the greatest improvement in low-SNR, multi-speaker environments, while AVSR embeddings alone perform best in noise-only scenarios. In addition, we develop a real-time streaming system that operates on a computer CPU and we provide a video demonstration and code repository. To our knowledge, this is the first open-source implementation of a real-time AVSE system.