Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment
This addresses the challenge of effective multimodal integration for emotion recognition in human-computer interaction, representing an incremental improvement over existing alignment strategies.
The paper tackled the problem of aligning speech and text features in multimodal emotion recognition by introducing a Multi-Granularity Cross-Modal Alignment framework, which outperformed state-of-the-art methods on the IEMOCAP dataset.
Multimodal emotion recognition (MER), leveraging speech and text, has emerged as a pivotal domain within human-computer interaction, demanding sophisticated methods for effective multimodal integration. The challenge of aligning features across these modalities is significant, with most existing approaches adopting a singular alignment strategy. Such a narrow focus not only limits model performance but also fails to address the complexity and ambiguity inherent in emotional expressions. In response, this paper introduces a Multi-Granularity Cross-Modal Alignment (MGCMA) framework, distinguished by its comprehensive approach encompassing distribution-based, instance-based, and token-based alignment modules. This framework enables a multi-level perception of emotional information across modalities. Our experiments on IEMOCAP demonstrate that our proposed method outperforms current state-of-the-art techniques.