IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation
This work addresses the problem of efficient and effective speech separation for applications like hearing aids or video conferencing by introducing a novel attention-based fusion method, though it is incremental over prior multimodal approaches.
The paper tackles audio-visual speech separation by proposing IIANet, a model that uses intra- and inter-attention mechanisms for multimodal feature fusion, achieving state-of-the-art performance on benchmarks like LRS2, LRS3, and VoxCeleb2 with improved efficiency, such as IIANet-fast having 7% of CTCNet's MACs and being 40% faster on CPUs.
Recent research has made significant progress in designing fusion modules for audio-visual speech separation. However, they predominantly focus on multi-modal fusion at a single temporal scale of auditory and visual features without employing selective attention mechanisms, which is in sharp contrast with the brain. To address this issue, We propose a novel model called Intra- and Inter-Attention Network (IIANet), which leverages the attention mechanism for efficient audio-visual feature fusion. IIANet consists of two types of attention blocks: intra-attention (IntraA) and inter-attention (InterA) blocks, where the InterA blocks are distributed at the top, middle and bottom of IIANet. Heavily inspired by the way how human brain selectively focuses on relevant content at various temporal scales, these blocks maintain the ability to learn modality-specific features and enable the extraction of different semantics from audio-visual features. Comprehensive experiments on three standard audio-visual separation benchmarks (LRS2, LRS3, and VoxCeleb2) demonstrate the effectiveness of IIANet, outperforming previous state-of-the-art methods while maintaining comparable inference time. In particular, the fast version of IIANet (IIANet-fast) has only 7% of CTCNet's MACs and is 40% faster than CTCNet on CPUs while achieving better separation quality, showing the great potential of attention mechanism for efficient and effective multimodal fusion.