Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition
This addresses the problem of robust emotion recognition in real-world video scenarios for AI applications, representing a novel method for a known bottleneck.
The paper tackles the problem of limited spatial and contextual awareness in Vision Large Language Models (VLLMs) for video-based emotion recognition by proposing Set-of-Vision-Text Prompting (SoVTP), which integrates spatial annotations, physiological signals, and contextual cues into a unified prompting strategy, achieving substantial improvements over existing visual prompting methods.
Vision Large Language Models (VLLMs) exhibit promising potential for multi-modal understanding, yet their application to video-based emotion recognition remains limited by insufficient spatial and contextual awareness. Traditional approaches, which prioritize isolated facial features, often neglect critical non-verbal cues such as body language, environmental context, and social interactions, leading to reduced robustness in real-world scenarios. To address this gap, we propose Set-of-Vision-Text Prompting (SoVTP), a novel framework that enhances zero-shot emotion recognition by integrating spatial annotations (e.g., bounding boxes, facial landmarks), physiological signals (facial action units), and contextual cues (body posture, scene dynamics, others' emotions) into a unified prompting strategy. SoVTP preserves holistic scene information while enabling fine-grained analysis of facial muscle movements and interpersonal dynamics. Extensive experiments show that SoVTP achieves substantial improvements over existing visual prompting methods, demonstrating its effectiveness in enhancing VLLMs' video emotion recognition capabilities.