Visual-Informed Speech Enhancement Using Attention-Based Beamforming
This work addresses the problem of suboptimal speech enhancement performance in challenging acoustic conditions for users of speech processing systems, offering an incremental improvement.
This paper proposes a Visual-Informed Neural Beamforming Network (VI-NBFNet) that integrates microphone array signal processing and deep neural networks with multimodal input features to improve speech enhancement. The system achieved better speech enhancement performance and robustness for both stationary and dynamic speaker scenarios compared to several baseline methods.
Recent studies have demonstrated that incorporating auxiliary information, such as speaker voiceprint or visual cues, can substantially improve Speech Enhancement (SE) performance. However, single-channel methods often yield suboptimal results in low signal-to-noise ratio (SNR) conditions, when there is high reverberation, or in complex scenarios involving dynamic speakers, overlapping speech, or non-stationary noise. To address these issues, we propose a novel Visual-Informed Neural Beamforming Network (VI-NBFNet), which integrates microphone array signal processing and deep neural networks (DNNs) using multimodal input features. The proposed network leverages a pretrained visual speech recognition model to extract lip movements as input features, which serve for voice activity detection (VAD) and target speaker identification. The system is intended to handle both static and moving speakers by introducing a supervised end-to-end beamforming framework equipped with an attention mechanism. The experimental results demonstrated that the proposed audiovisual system has achieved better SE performance and robustness for both stationary and dynamic speaker scenarios, compared to several baseline methods.