Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition
This work addresses robust speech recognition for noisy settings, offering incremental improvements by focusing on video features, which are often overlooked in prior methods.
The paper tackled the problem of audio-visual speech recognition in noisy environments by strengthening video features through learning temporal dynamics and using cross-modal attention, achieving state-of-the-art performance on LRS2 and LRS3 benchmarks, particularly excelling in babble and speech noise scenarios.
Audio-visual speech recognition (AVSR) aims to transcribe human speech using both audio and video modalities. In practical environments with noise-corrupted audio, the role of video information becomes crucial. However, prior works have primarily focused on enhancing audio features in AVSR, overlooking the importance of video features. In this study, we strengthen the video features by learning three temporal dynamics in video data: context order, playback direction, and the speed of video frames. Cross-modal attention modules are introduced to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics. Based on our approach, we achieve the state-of-the-art performance on the LRS2 and LRS3 AVSR benchmarks for the noise-dominant settings. Our approach excels in scenarios especially for babble and speech noise, indicating the ability to distinguish the speech signal that should be recognized from lip movements in the video modality. We support the validity of our methodology by offering the ablation experiments for the temporal dynamics losses and the cross-modal attention architecture design.