Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations
This work improves emotion recognition accuracy for multimodal conversational systems, representing an incremental advance in handling long dialogues and aligning text, audio, and video features.
The paper tackled the problem of Emotion Recognition in Conversations by addressing challenges in multimodal alignment and contextual information capture, proposing the MaTAV network, which significantly outperformed existing state-of-the-art methods on MELD and IEMOCAP datasets.
Emotion Recognition in Conversations (ERCs) is a vital area within multimodal interaction research, dedicated to accurately identifying and classifying the emotions expressed by speakers throughout a conversation. Traditional ERC approaches predominantly rely on unimodal cues\-such as text, audio, or visual data\-leading to limitations in their effectiveness. These methods encounter two significant challenges: 1) Consistency in multimodal information. Before integrating various modalities, it is crucial to ensure that the data from different sources is aligned and coherent. 2) Contextual information capture. Successfully fusing multimodal features requires a keen understanding of the evolving emotional tone, especially in lengthy dialogues where emotions may shift and develop over time. To address these limitations, we propose a novel Mamba-enhanced Text-Audio-Video alignment network (MaTAV) for the ERC task. MaTAV is with the advantages of aligning unimodal features to ensure consistency across different modalities and handling long input sequences to better capture contextual multimodal information. The extensive experiments on the MELD and IEMOCAP datasets demonstrate that MaTAV significantly outperforms existing state-of-the-art methods on the ERC task with a big margin.