Multi-Modal Emotion Detection with Transfer Learning
This work aims to improve the generalizability of emotion detection systems for researchers and developers working with limited and diverse speech datasets, offering an incremental improvement.
This paper tackles the challenge of multi-modal emotion detection in speech, which is hampered by small and inconsistently labeled datasets. The authors propose a transfer learning approach using a TDNN for speech and BERT for text, followed by a pLDA classifier. Their best model achieves an Equal Error Rate (EER) of 38.05% on IEMOCAP when trained on VoxCeleb and Crema-D, and 25.72% when a portion of IEMOCAP is included in training.
Automated emotion detection in speech is a challenging task due to the complex interdependence between words and the manner in which they are spoken. It is made more difficult by the available datasets; their small size and incompatible labeling idiosyncrasies make it hard to build generalizable emotion detection systems. To address these two challenges, we present a multi-modal approach that first transfers learning from related tasks in speech and text to produce robust neural embeddings and then uses these embeddings to train a pLDA classifier that is able to adapt to previously unseen emotions and domains. We begin by training a multilayer TDNN on the task of speaker identification with the VoxCeleb corpora and then fine-tune it on the task of emotion identification with the Crema-D corpus. Using this network, we extract speech embeddings for Crema-D from each of its layers, generate and concatenate text embeddings for the accompanying transcripts using a fine-tuned BERT model and then train an LDA - pLDA classifier on the resulting dense representations. We exhaustively evaluate the predictive power of every component: the TDNN alone, speech embeddings from each of its layers alone, text embeddings alone and every combination thereof. Our best variant, trained on only VoxCeleb and Crema-D and evaluated on IEMOCAP, achieves an EER of 38.05%. Including a portion of IEMOCAP during training produces a 5-fold averaged EER of 25.72% (For comparison, 44.71% of the gold-label annotations include at least one annotator who disagrees).