Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models
This work addresses cross-lingual speech emotion recognition for applications in multilingual AI systems, but it is incremental as it builds on existing self-supervised learning methods.
The study compared human and self-supervised learning models in cross-lingual speech emotion recognition, finding that models with knowledge transfer can match native speaker performance and that dialect significantly affects recognition for those without linguistic background.
Utilizing Self-Supervised Learning (SSL) models for Speech Emotion Recognition (SER) has proven effective, yet limited research has explored cross-lingual scenarios. This study presents a comparative analysis between human performance and SSL models, beginning with a layer-wise analysis and an exploration of parameter-efficient fine-tuning strategies in monolingual, cross-lingual, and transfer learning contexts. We further compare the SER ability of models and humans at both utterance- and segment-levels. Additionally, we investigate the impact of dialect on cross-lingual SER through human evaluation. Our findings reveal that models, with appropriate knowledge transfer, can adapt to the target language and achieve performance comparable to native speakers. We also demonstrate the significant effect of dialect on SER for individuals without prior linguistic and paralinguistic background. Moreover, both humans and models exhibit distinct behaviors across different emotions. These results offer new insights into the cross-lingual SER capabilities of SSL models, underscoring both their similarities to and differences from human emotion perception.