Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning
This addresses a difficult problem in affective computing for applications requiring nuanced emotion understanding from visual and contextual data, but it appears incremental as it builds on existing models.
The paper tackled the emotional theory of mind problem by constructing narrative captions from images using methods like CLIP and LLaVA, and using these to guide language models like GPT-4 for emotion reasoning, showing this combination is promising for the task.
The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing "narrative captions" relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on "Who", "What", "Where" and "How" questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLaVa, GPT-Vision). We evaluate the use of the resulting models in an image-to-language-to-emotion task. Our experiments showed that combining the "Fast" narrative descriptors and "Slow" reasoning of language models is a promising way to achieve emotional theory of mind.