Gaze-enhanced Crossmodal Embeddings for Emotion Recognition
This work addresses the challenge of improving emotion recognition accuracy for applications like human-computer interaction by integrating gaze, though it is incremental as it builds on existing crossmodal frameworks.
The paper tackled the problem of emotion recognition by incorporating an explicit gaze representation into a crossmodal embedding framework, resulting in outperforming previous state-of-the-art methods for audio-only and video-only classification on the One-Minute Gradual Emotion Recognition dataset.
Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e.g. speech during a phone call. While previous work proposed crossmodal emotion embeddings to improve monomodal recognition performance, despite its importance, an explicit representation of gaze was not included. We propose a new approach to emotion recognition that incorporates an explicit representation of gaze in a crossmodal emotion embedding framework. We show that our method outperforms the previous state of the art for both audio-only and video-only emotion classification on the popular One-Minute Gradual Emotion Recognition dataset. Furthermore, we report extensive ablation experiments and provide detailed insights into the performance of different state-of-the-art gaze representations and integration strategies. Our results not only underline the importance of gaze for emotion recognition but also demonstrate a practical and highly effective approach to leveraging gaze information for this task.