Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition
This work addresses the challenge of effectively combining audio and video data for emotion recognition, which is an incremental improvement in a domain-specific area.
The paper tackles the problem of inter-modal information fusion in multimodal emotion recognition by proposing Foal-Net, a framework that uses alignment and label matching to enhance fusion, resulting in state-of-the-art performance on the IEMOCAP corpus.
To address the limitation in multimodal emotion recognition (MER) performance arising from inter-modal information fusion, we propose a novel MER framework based on multitask learning where fusion occurs after alignment, called Foal-Net. The framework is designed to enhance the effectiveness of modality fusion and includes two auxiliary tasks: audio-video emotion alignment (AVEL) and cross-modal emotion label matching (MEM). First, AVEL achieves alignment of emotional information in audio-video representations through contrastive learning. Then, a modal fusion network integrates the aligned features. Meanwhile, MEM assesses whether the emotions of the current sample pair are the same, providing assistance for modal information fusion and guiding the model to focus more on emotional information. The experimental results conducted on IEMOCAP corpus show that Foal-Net outperforms the state-of-the-art methods and emotion alignment is necessary before modal fusion.