Modality Attention for End-to-End Audio-visual Speech Recognition
This addresses the problem of noisy speech recognition for users in challenging acoustic environments, representing an incremental advance over traditional fusion methods.
The paper tackles robust speech recognition in noisy environments by proposing a modality attention method for audio-visual speech recognition, achieving relative improvements of 2% to 36% over audio-only methods depending on noise levels.
Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for robust speech recognition, especially in noisy environment. In this paper, we propose a novel multimodal attention based method for audio-visual speech recognition which could automatically learn the fused representation from both modalities based on their importance. Our method is realized using state-of-the-art sequence-to-sequence (Seq2seq) architectures. Experimental results show that relative improvements from 2% up to 36% over the auditory modality alone are obtained depending on the different signal-to-noise-ratio (SNR). Compared to the traditional feature concatenation methods, our proposed approach can achieve better recognition performance under both clean and noisy conditions. We believe modality attention based end-to-end method can be easily generalized to other multimodal tasks with correlated information.