Multi-Task Learning Framework for Emotion Recognition in-the-wild
This work addresses emotion recognition for affective computing applications, but it is incremental as it builds on existing methods like MAE, transformers, and multi-task learning for a specific competition challenge.
The paper tackled emotion recognition in-the-wild by proposing a multi-task learning framework that integrates visual feature representation, temporal encoding, and task correlation modeling, achieving first place in the ABAW4 competition with validation and test scores of 1.7607 and 1.4361, respectively.
This paper presents our system for the Multi-Task Learning (MTL) Challenge in the 4th Affective Behavior Analysis in-the-wild (ABAW) competition. We explore the research problems of this challenge from three aspects: 1) For obtaining efficient and robust visual feature representations, we propose MAE-based unsupervised representation learning and IResNet/DenseNet-based supervised representation learning methods; 2) Considering the importance of temporal information in videos, we explore three types of sequential encoders to capture the temporal information, including the encoder based on transformer, the encoder based on LSTM, and the encoder based on GRU; 3) For modeling the correlation between these different tasks (i.e., valence, arousal, expression, and AU) for multi-task affective analysis, we first explore the dependency between these different tasks and propose three multi-task learning frameworks to model the correlations effectively. Our system achieves the performance of $1.7607$ on the validation dataset and $1.4361$ on the test dataset, ranking first in the MTL Challenge. The code is available at https://github.com/AIM3-RUC/ABAW4.