Evaluating Vision-Language Models for Emotion Recognition
This research addresses the problem of improving emotion recognition in VLMs, which is crucial for effective human-computer interaction, particularly for applications that require empathetic communication.
This study evaluated the performance of Vision-Language Models (VLMs) in recognizing evoked emotions from images, identifying key factors that impact performance and characterizing errors made by VLMs, with the goal of informing future fine-tuning efforts. The study found that VLMs' performance on emotion recognition tasks depends on several important factors.
Large Vision-Language Models (VLMs) have achieved unprecedented success in several objective multimodal reasoning tasks. However, to further enhance their capabilities of empathetic and effective communication with humans, improving how VLMs process and understand emotions is crucial. Despite significant research attention on improving affective understanding, there is a lack of detailed evaluations of VLMs for emotion-related tasks, which can potentially help inform downstream fine-tuning efforts. In this work, we present the first comprehensive evaluation of VLMs for recognizing evoked emotions from images. We create a benchmark for the task of evoked emotion recognition and study the performance of VLMs for this task, from perspectives of correctness and robustness. Through several experiments, we demonstrate important factors that emotion recognition performance depends on, and also characterize the various errors made by VLMs in the process. Finally, we pinpoint potential causes for errors through a human evaluation study. We use our experimental results to inform recommendations for the future of emotion research in the context of VLMs.