UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause
This work addresses the problem of integrating emotion and cause analysis in multimodal conversations for researchers in affective computing, though it is incremental as it builds on existing tasks.
The paper tackles the separate treatment of multimodal emotion recognition and emotion cause extraction by proposing UniMEEC, a unified framework that models their causality, achieving consistent improvements over state-of-the-art methods on four benchmark datasets.
Multimodal emotion recognition in conversation (MERC) and multimodal emotion-cause pair extraction (MECPE) have recently garnered significant attention. Emotions are the expression of affect or feelings; responses to specific events, or situations -- known as emotion causes. Both collectively explain the causality between human emotion and intents. However, existing works treat emotion recognition and emotion cause extraction as two individual problems, ignoring their natural causality. In this paper, we propose a Unified Multimodal Emotion recognition and Emotion-Cause analysis framework (UniMEEC) to explore the causality between emotion and emotion cause. Concretely, UniMEEC reformulates the MERC and MECPE tasks as mask prediction problems and unifies them with a causal prompt template. To differentiate the modal effects, UniMEEC proposes a multimodal causal prompt to probe the pre-trained knowledge specified to modality and implements cross-task and cross-modality interactions under task-oriented settings. Experiment results on four public benchmark datasets verify the model performance on MERC and MECPE tasks and achieve consistent improvements compared with the previous state-of-the-art methods.