Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition
This work addresses the challenge of recognizing subtle emotions in real-life speeches, which is important for applications like human-computer interaction, but it appears incremental as it builds on existing multimodal methods.
The authors tackled fine-grained speech emotion recognition by proposing a multimodal deep learning approach with temporal alignment mean-max pooling and a cross modality excitement module, achieving superior prediction accuracy on two real-world datasets compared to a wide range of baselines.
Speech emotion recognition is a challenging task because the emotion expression is complex, multimodal and fine-grained. In this paper, we propose a novel multimodal deep learning approach to perform fine-grained emotion recognition from real-life speeches. We design a temporal alignment mean-max pooling mechanism to capture the subtle and fine-grained emotions implied in every utterance. In addition, we propose a cross modality excitement module to conduct sample-specific adjustment on cross modality embeddings and adaptively recalibrate the corresponding values by its aligned latent features from the other modality. Our proposed model is evaluated on two well-known real-world speech emotion recognition datasets. The results demonstrate that our approach is superior on the prediction tasks for multimodal speech utterances, and it outperforms a wide range of baselines in terms of prediction accuracy. Further more, we conduct detailed ablation studies to show that our temporal alignment mean-max pooling mechanism and cross modality excitement significantly contribute to the promising results. In order to encourage the research reproducibility, we make the code publicly available at \url{https://github.com/tal-ai/FG_CME.git}.