BeMERC: Behavior-Aware MLLM-based Framework for Multimodal Emotion Recognition in Conversation
This addresses the problem of more accurate emotion recognition in conversations for developing empathetic machines, though it is incremental by adding behavior data to existing MLLM approaches.
The paper tackles multimodal emotion recognition in conversation by incorporating video-derived behavior information (facial micro-expressions, body language, posture) into an MLLM-based framework, achieving superior performance over state-of-the-art methods on two benchmark datasets.
Multimodal emotion recognition in conversation (MERC), the task of identifying the emotion label for each utterance in a conversation, is vital for developing empathetic machines. Current MLLM-based MERC studies focus mainly on capturing the speaker's textual or vocal characteristics, but ignore the significance of video-derived behavior information. Different from text and audio inputs, learning videos with rich facial expression, body language and posture, provides emotion trigger signals to the models for more accurate emotion predictions. In this paper, we propose a novel behavior-aware MLLM-based framework (BeMERC) to incorporate speaker's behaviors, including subtle facial micro-expression, body language and posture, into a vanilla MLLM-based MERC model, thereby facilitating the modeling of emotional dynamics during a conversation. Furthermore, BeMERC adopts a two-stage instruction tuning strategy to extend the model to the conversations scenario for end-to-end training of a MERC predictor. Experiments demonstrate that BeMERC achieves superior performance than the state-of-the-art methods on two benchmark datasets, and also provides a detailed discussion on the significance of video-derived behavior information in MERC.