Multimodal Video Emotion Recognition with Reliable Reasoning Priors
This work improves emotion recognition accuracy for applications like human-computer interaction, though it is incremental as it builds on existing multimodal and MLLM approaches.
The study tackled multimodal emotion recognition by integrating reliable reasoning priors from MLLMs and addressing class imbalance with a balanced dual-contrastive learning loss, achieving substantial performance gains on the MER2024 benchmark.
This study investigates the integration of trustworthy prior reasoning knowledge from MLLMs into multimodal emotion recognition. We employ Gemini to generate fine-grained, modality-separable reasoning traces, which are injected as priors during the fusion stage to enrich cross-modal interactions. To mitigate the pronounced class-imbalance in multimodal emotion recognition, we introduce Balanced Dual-Contrastive Learning, a loss formulation that jointly balances inter-class and intra-class distributions. Applied to the MER2024 benchmark, our prior-enhanced framework yields substantial performance gains, demonstrating that the reliability of MLLM-derived reasoning can be synergistically combined with the domain adaptability of lightweight fusion networks for robust, scalable emotion recognition.