Multi Modal Facial Expression Recognition with Transformer-Based Fusion Networks and Dynamic Sampling
This work addresses emotion detection for applications like mental health analysis, but it is incremental as it builds on existing transformer methods with multi-modal fusion.
The paper tackles facial expression recognition by combining audio and visual data to resolve ambiguous expressions, achieving competitive results on the ABAW challenge dataset.
Facial expression recognition is an essential task for various applications, including emotion detection, mental health analysis, and human-machine interactions. In this paper, we propose a multi-modal facial expression recognition method that exploits audio information along with facial images to provide a crucial clue to differentiate some ambiguous facial expressions. Specifically, we introduce a Modal Fusion Module (MFM) to fuse audio-visual information, where image and audio features are extracted from Swin Transformer. Additionally, we tackle the imbalance problem in the dataset by employing dynamic data resampling. Our model has been evaluated in the Affective Behavior in-the-wild (ABAW) challenge of CVPR 2023.