On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognition
This addresses the need for more robust conversational agents in noisy real-world settings, though it is incremental as it builds on existing multitask learning methods.
The paper tackled the problem of improving speech emotion recognition (SER) and automatic speech recognition (ASR) in noisy environments by proposing a joint multitask learning approach, showing improvements of 10.7% in ASR word error rate and 2.3% in SER accuracy in clean scenarios and better noise robustness.
New-age conversational agent systems perform both speech emotion recognition (SER) and automatic speech recognition (ASR) using two separate and often independent approaches for real-world application in noisy environments. In this paper, we investigate a joint ASR-SER multitask learning approach in a low-resource setting and show that improvements are observed not only in SER, but also in ASR. We also investigate the robustness of such jointly trained models to the presence of background noise, babble, and music. Experimental results on the IEMOCAP dataset show that joint learning can improve ASR word error rate (WER) and SER classification accuracy by 10.7% and 2.3% respectively in clean scenarios. In noisy scenarios, results on data augmented with MUSAN show that the joint approach outperforms the independent ASR and SER approaches across many noisy conditions. Overall, the joint ASR-SER approach yielded more noise-resistant models than the independent ASR and SER approaches.