Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning
This addresses a specific issue in multimodal emotion reasoning for AI applications, but it is incremental as it builds on existing MLLM methods to handle emotion conflicts.
The paper tackles the problem of multimodal emotion reasoning models overlooking emotion conflicts where cues from different modalities are inconsistent, and it introduces a new benchmark CA-MER and a framework MoSEAR that achieves state-of-the-art performance, with experiments showing improvements on multiple benchmarks including MER2023, EMER, DFEW, and CA-MER.
Despite their strong performance in multimodal emotion reasoning, existing Multimodal Large Language Models (MLLMs) often overlook the scenarios involving emotion conflicts, where emotional cues from different modalities are inconsistent. To fill this gap, we first introduce CA-MER, a new benchmark designed to examine MLLMs under realistic emotion conflicts. It consists of three subsets: video-aligned, audio-aligned, and consistent, where only one or all modalities reflect the true emotion. However, evaluations on our CA-MER reveal that current state-of-the-art emotion MLLMs systematically over-rely on audio signal during emotion conflicts, neglecting critical cues from visual modality. To mitigate this bias, we propose MoSEAR, a parameter-efficient framework that promotes balanced modality integration. MoSEAR consists of two modules: (1)MoSE, modality-specific experts with a regularized gating mechanism that reduces modality bias in the fine-tuning heads; and (2)AR, an attention reallocation mechanism that rebalances modality contributions in frozen backbones during inference. Our framework offers two key advantages: it mitigates emotion conflicts and improves performance on consistent samples-without incurring a trade-off between audio and visual modalities. Experiments on multiple benchmarks-including MER2023, EMER, DFEW, and our CA-MER-demonstrate that MoSEAR achieves state-of-the-art performance, particularly under modality conflict conditions.