Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
This addresses the problem of high annotation costs and label ambiguity in speech emotion recognition for researchers and practitioners, offering a novel transfer learning approach.
The paper tackles the challenge of training speech emotion recognition models without labeled audio data by using cross-modal distillation from facial expression annotations, achieving state-of-the-art performance on a standard benchmark.
Obtaining large, human labelled speech datasets to train models for emotion recognition is a notoriously challenging task, hindered by annotation cost and label ambiguity. In this work, we consider the task of learning embeddings for speech classification without access to any form of labelled audio. We base our approach on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker. By exploiting this relationship, we show that annotations of expression can be transferred from the visual domain (faces) to the speech domain (voices) through cross-modal distillation. We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets. Code, models and data are available.