Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models
This work addresses efficiency for speech emotion recognition systems by reducing model size while maintaining performance, which is incremental as it builds on existing SSL methods.
The paper tackled the problem of large model sizes hindering practical self-supervised learning (SSL) implementations in speech emotion recognition (SER) by proposing an audio-textual distilled SSL framework. It achieved on par performance across arousal, valence, and dominance dimensions with only ~20% of the trainable parameters of a large SSL model on the MSP-Podcast v1.10 dataset.
In large part due to their implicit semantic modeling, self-supervised learning (SSL) methods have significantly increased the performance of valence recognition in speech emotion recognition (SER) systems. Yet, their large size may often hinder practical implementations. In this work, we take HuBERT as an example of an SSL model and analyze the relevance of each of its layers for SER. We show that shallow layers are more important for arousal recognition while deeper layers are more important for valence. This observation motivates the importance of additional textual information for accurate valence recognition, as the distilled framework lacks the depth of its large-scale SSL teacher. Thus, we propose an audio-textual distilled SSL framework that, while having only ~20% of the trainable parameters of a large SSL model, achieves on par performance across the three emotion dimensions (arousal, valence, dominance) on the MSP-Podcast v1.10 dataset.