Gems: Group Emotion Profiling Through Multimodal Situational Understanding
This work addresses the need for holistic emotion analysis in social contexts, but it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of understanding emotions at individual, group, and event levels in multi-person social situations by proposing GEMS, a multimodal framework that predicts fine-grained emotions, achieving effectiveness demonstrated through quantitative and qualitative comparisons on the VGAF-GEMS benchmark.
Understanding individual, group and event level emotions along with contextual information is crucial for analyzing a multi-person social situation. To achieve this, we frame emotion comprehension as the task of predicting fine-grained individual emotion to coarse grained group and event level emotion. We introduce GEMS that leverages a multimodal swin-transformer and S3Attention based architecture, which processes an input scene, group members, and context information to generate joint predictions. Existing multi-person emotion related benchmarks mainly focus on atomic interactions primarily based on emotion perception over time and group level. To this end, we extend and propose VGAF-GEMS to provide more fine grained and holistic analysis on top of existing group level annotation of VGAF dataset. GEMS aims to predict basic discrete and continuous emotions (including valence and arousal) as well as individual, group and event level perceived emotions. Our benchmarking effort links individual, group and situational emotional responses holistically. The quantitative and qualitative comparisons with adapted state-of-the-art models demonstrate the effectiveness of GEMS framework on VGAF-GEMS benchmarking. We believe that it will pave the way of further research. The code and data is available at: https://github.com/katariaak579/GEMS