Deep Fusion: An Attention Guided Factorized Bilinear Pooling for Audio-video Emotion Recognition
This work addresses emotion recognition for human-computer interaction by improving multimodal fusion, though it is incremental as it builds on existing bilinear pooling methods.
The paper tackled audio-video emotion recognition by introducing factorized bilinear pooling with attention to capture complex associations between modalities, achieving an accuracy of 62.48% on the AFEW database, outperforming state-of-the-art results.
Automatic emotion recognition (AER) is a challenging task due to the abstract concept and multiple expressions of emotion. Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems. Inspired by this cognitive process in human beings, it's natural to simultaneously utilize audio and visual information in AER. However, most traditional fusion approaches only build a linear paradigm, such as feature concatenation and multi-system fusion, which hardly captures complex association between audio and video. In this paper, we introduce factorized bilinear pooling (FBP) to deeply integrate the features of audio and video. Specifically, the features are selected through the embedded attention mechanism from respective modalities to obtain the emotion-related regions. The whole pipeline can be completed in a neural network. Validated on the AFEW database of the audio-video sub-challenge in EmotiW2018, the proposed approach achieves an accuracy of 62.48%, outperforming the state-of-the-art result.