Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier
This addresses the need for emotionally coherent and trustworthy human-computer interaction systems, though it is incremental as it builds on existing MLLM frameworks without architectural changes.
The paper tackled the problem of multimodal large language models generating emotion explanations that diverge from predicted emotions, proposing an Emotional Rationale Verifier and Explanation Reward to improve consistency. The method significantly enhanced faithful explanation-prediction consistency and explanation emotion accuracy on the MAFW and DFEW datasets.
The recent advancement of Multimodal Large Language Models (MLLMs) is transforming human-computer interaction (HCI) from surface-level exchanges into more nuanced and emotionally intelligent communication. To realize this shift, emotion understanding becomes essential allowing systems to capture subtle cues underlying user intent. Furthermore, providing faithful explanations for predicted emotions is crucial to ensure interpretability and build user trust. However, current MLLM-based methods often generate emotion explanations that diverge from the target labels and sometimes even contradict their own predicted emotions. This inconsistency poses a critical risk for misunderstanding and erodes reliability in interactive settings. To address this, we propose a novel approach: the Emotional Rationale Verifier (ERV) and an Explanation Reward. Our method guides the model to produce reasoning that is explicitly consistent with the target emotion during multimodal emotion recognition without modifying the model architecture or requiring additional paired video-description annotations. Our method significantly improves faithful explanation-prediction consistency and explanation emotion accuracy on the MAFW and DFEW datasets. Through extensive experiments and human evaluations, we show that our approach not only enhances alignment between explanation and prediction but also empowers MLLMs to deliver emotionally coherent, trustworthy interactions, marking a key step toward truly human-like HCI systems.