How Speech is Recognized to Be Emotional - A Study Based on Information Decomposition
This work addresses the challenge of generalizability in speech emotion recognition for AI applications, but it is incremental as it builds on existing decomposition methods.
The study investigated the impact of decomposed speech components (content, pitch, rhythm) on emotion recognition systems, finding that rhythm is the most important factor and that removing unimportant components can improve cross-corpus generalization, though current models perform poorly across datasets.
The way that humans encode their emotion into speech signals is complex. For instance, an angry man may increase his pitch and speaking rate, and use impolite words. In this paper, we present a preliminary study on various emotional factors and investigate how each of them impacts modern emotion recognition systems. The key tool of our study is the SpeechFlow model presented recently, by which we are able to decompose speech signals into separate information factors (content, pitch, rhythm). Based on this decomposition, we carefully studied the performance of each information component and their combinations. We conducted the study on three different speech emotion corpora and chose an attention-based convolutional RNN as the emotion classifier. Our results show that rhythm is the most important component for emotional expression. Moreover, the cross-corpus results are very bad (even worse than guess), demonstrating that the present speech emotion recognition model is rather weak. Interestingly, by removing one or several unimportant components, the cross-corpus results can be improved. This demonstrates the potential of the decomposition approach towards a generalizable emotion recognition.