Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition
This work addresses speech recognition in noisy scenarios for applications like communication systems, but it is incremental as it builds on existing transformer-based audio-visual fusion methods.
The paper tackled the problem of robust speech recognition in noisy environments by proposing an attentive fusion block integrated into the encoding process, which increased recognition rates by 0.55%, 4.51%, and 4.61% on average under clean, seen, and unseen noise conditions compared to the state-of-the-art.
Audio-visual information fusion enables a performance improvement in speech recognition performed in complex acoustic scenarios, e.g., noisy environments. It is required to explore an effective audio-visual fusion strategy for audiovisual alignment and modality reliability. Different from the previous end-to-end approaches where the audio-visual fusion is performed after encoding each modality, in this paper we propose to integrate an attentive fusion block into the encoding process. It is shown that the proposed audio-visual fusion method in the encoder module can enrich audio-visual representations, as the relevance between the two modalities is leveraged. In line with the transformer-based architecture, we implement the embedded fusion block using a multi-head attention based audiovisual fusion with one-way or two-way interactions. The proposed method can sufficiently combine the two streams and weaken the over-reliance on the audio modality. Experiments on the LRS3-TED dataset demonstrate that the proposed method can increase the recognition rate by 0.55%, 4.51% and 4.61% on average under the clean, seen and unseen noise conditions, respectively, compared to the state-of-the-art approach.