Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention
This work addresses the problem of improving sentiment analysis accuracy by reducing misunderstandings from ambiguity and sarcasm for applications in human-computer interaction.
The paper tackled the challenge of fusing multimodal features for sentiment analysis by designing a novel multi-head attention network that assigns different attention to pairwise modality interactions, and it outperformed existing methods on four public datasets.
Humans express feelings or emotions via different channels. Take language as an example, it entails different sentiments under different visual-acoustic contexts. To precisely understand human intentions as well as reduce the misunderstandings caused by ambiguity and sarcasm, we should consider multimodal signals including textual, visual and acoustic signals. The crucial challenge is to fuse different modalities of features for sentiment analysis. To effectively fuse the information carried by different modalities and better predict the sentiments, we design a novel multi-head attention based fusion network, which is inspired by the observations that the interactions between any two pair-wise modalities are different and they do not equally contribute to the final sentiment prediction. By assigning the acoustic-visual, acoustic-textual and visual-textual features with reasonable attention and exploiting a residual structure, we attend to attain the significant features. We conduct extensive experiments on four public multimodal datasets including one in Chinese and three in English. The results show that our approach outperforms the existing methods and can explain the contributions of bimodal interaction in multiple modalities.