Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues
This work addresses robust speech enhancement for applications like hearing aids or noisy environments by using facial cues, though it appears incremental as it builds on existing audio-visual methods with attention mechanisms.
The paper tackles audio-visual speech enhancement by leveraging facial cues beyond just lip movements, proposing a Dual Attention Cooperative Framework (DualAVSE) that ignores speech-unrelated information and dynamically integrates visual and audio features. The model consistently outperforms existing methods across multiple metrics on various datasets, including challenging cases with unreliable visual information.
In this work, we focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE). The facial region, encompassing the lip region, reflects additional speech-related attributes such as gender, skin color, nationality, etc., which contribute to the effectiveness of AVSE. However, static and dynamic speech-unrelated attributes also exist, causing appearance changes during speech. To address these challenges, we propose a Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated information, capture speech-related information with facial cues, and dynamically integrate it with the audio signal for AVSE. Specifically, we introduce a spatial attention-based visual encoder to capture and enhance visual speech information beyond the lip region, incorporating global facial context and automatically ignoring speech-unrelated information for robust visual feature extraction. Additionally, a dynamic visual feature fusion strategy is introduced by integrating a temporal-dimensional self-attention module, enabling the model to robustly handle facial variations. The acoustic noise in the speaking process is variable, impacting audio quality. Therefore, a dynamic fusion strategy for both audio and visual features is introduced to address this issue. By integrating cooperative dual attention in the visual encoder and audio-visual fusion strategy, our model effectively extracts beneficial speech information from both audio and visual cues for AVSE. Thorough analysis and comparison on different datasets, including normal and challenging cases with unreliable or absent visual information, consistently show our model outperforming existing methods across multiple metrics.