Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
This work addresses emotion prediction for affective computing applications, but it is incremental as it builds on existing fusion strategies with a novel attention mechanism.
The paper tackled continuous dimensional emotion prediction by proposing a conditional attention fusion strategy that dynamically weights modalities at each time step, outperforming common fusion methods on the AVEC2015 benchmark for valence prediction.
Continuous dimensional emotion prediction is a challenging task where the fusion of various modalities usually achieves state-of-the-art performance such as early fusion or late fusion. In this paper, we propose a novel multi-modal fusion strategy named conditional attention fusion, which can dynamically pay attention to different modalities at each time step. Long-short term memory recurrent neural networks (LSTM-RNN) is applied as the basic uni-modality model to capture long time dependencies. The weights assigned to different modalities are automatically decided by the current input features and recent history information rather than being fixed at any kinds of situation. Our experimental results on a benchmark dataset AVEC2015 show the effectiveness of our method which outperforms several common fusion strategies for valence prediction.