Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos
This work addresses emotion analysis for human-computer interaction applications, but it is incremental as it builds on existing multimodal approaches for a specific competition.
The paper tackles emotion estimation in videos by developing an attention-based multimodal fusion framework using audio, pose, and image features from the ABAW competition dataset, achieving a performance score of 0.361 on the validation set.
The continuous improvement of human-computer interaction technology makes it possible to compute emotions. In this paper, we introduce our submission to the CVPR 2023 Competition on Affective Behavior Analysis in-the-wild (ABAW). Sentiment analysis in human-computer interaction should, as far as possible Start with multiple dimensions, fill in the single imperfect emotion channel, and finally determine the emotion tendency by fitting multiple results. Therefore, We exploited multimodal features extracted from video of different lengths from the competition dataset, including audio, pose and images. Well-informed emotion representations drive us to propose a Attention-based multimodal framework for emotion estimation. Our system achieves the performance of 0.361 on the validation dataset. The code is available at [https://github.com/xkwangcn/ABAW-5th-RT-IAI].