LanSER: Language-Model Supported Speech Emotion Recognition
This addresses the challenge of costly human-labeled data for SER, making it easier to scale to large datasets and nuanced emotion taxonomies, though it is an incremental improvement over existing weakly-supervised approaches.
The authors tackled the problem of scaling speech emotion recognition (SER) by developing LanSER, a method that uses pre-trained large language models to infer weak emotion labels from unlabeled speech data via textual entailment, enabling models to outperform baselines on standard datasets and improve label efficiency.
Speech emotion recognition (SER) models typically rely on costly human-labeled data for training, making scaling methods to large speech datasets and nuanced emotion taxonomies difficult. We present LanSER, a method that enables the use of unlabeled data by inferring weak emotion labels via pre-trained large language models through weakly-supervised learning. For inferring weak labels constrained to a taxonomy, we use a textual entailment approach that selects an emotion label with the highest entailment score for a speech transcript extracted via automatic speech recognition. Our experimental results show that models pre-trained on large datasets with this weak supervision outperform other baseline models on standard SER datasets when fine-tuned, and show improved label efficiency. Despite being pre-trained on labels derived only from text, we show that the resulting representations appear to model the prosodic content of speech.