Fusing ASR Outputs in Joint Training for Speech Emotion Recognition
This work addresses the problem of improving SER accuracy for applications like human-computer interaction by integrating ASR features, though it is incremental as it builds on existing methods for combining acoustic and linguistic information.
The paper tackles the challenge of obtaining reliable linguistic features for Speech Emotion Recognition (SER) by proposing a joint training approach that fuses Automatic Speech Recognition (ASR) outputs, achieving 63.4% weighted accuracy on the IEMOCAP corpus, close to results using ground-truth transcripts.
Alongside acoustic information, linguistic features based on speech transcripts have been proven useful in Speech Emotion Recognition (SER). However, due to the scarcity of emotion labelled data and the difficulty of recognizing emotional speech, it is hard to obtain reliable linguistic features and models in this research area. In this paper, we propose to fuse Automatic Speech Recognition (ASR) outputs into the pipeline for joint training SER. The relationship between ASR and SER is understudied, and it is unclear what and how ASR features benefit SER. By examining various ASR outputs and fusion methods, our experiments show that in joint ASR-SER training, incorporating both ASR hidden and text output using a hierarchical co-attention fusion approach improves the SER performance the most. On the IEMOCAP corpus, our approach achieves 63.4% weighted accuracy, which is close to the baseline results achieved by combining ground-truth transcripts. In addition, we also present novel word error rate analysis on IEMOCAP and layer-difference analysis of the Wav2vec 2.0 model to better understand the relationship between ASR and SER.