An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild
This work addresses emotion recognition in uncontrolled environments for applications like human-computer interaction, but it is incremental as it builds on existing multi-modal methods with a focus on under-explored features.
The paper tackles video-based audio-visual emotion recognition in-the-wild by leveraging bodily and contextual features to address challenges like poor illumination and low resolution, with experiments on the Aff-Wild2 dataset verifying the approach's validity.
In this work we tackle the task of video-based audio-visual emotion recognition, within the premises of the 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW2). Poor illumination conditions, head/body orientation and low image resolution constitute factors that can potentially hinder performance in case of methodologies that solely rely on the extraction and analysis of facial features. In order to alleviate this problem, we leverage both bodily and contextual features, as part of a broader emotion recognition framework. We choose to use a standard CNN-RNN cascade as the backbone of our proposed model for sequence-to-sequence (seq2seq) learning. Apart from learning through the RGB input modality, we construct an aural stream which operates on sequences of extracted mel-spectrograms. Our extensive experiments on the challenging and newly assembled Aff-Wild2 dataset verify the validity of our intuitive multi-stream and multi-modal approach towards emotion recognition in-the-wild. Emphasis is being laid on the the beneficial influence of the human body and scene context, as aspects of the emotion recognition process that have been left relatively unexplored up to this point. All the code was implemented using PyTorch and is publicly available.