Frame-level emotional state alignment method for speech emotion recognition
This work addresses a specific bottleneck in speech emotion recognition for human-computer interaction, but it is incremental as it builds on existing pretrained models and clustering techniques.
The authors tackled the problem of frame-level inconsistency in speech emotion recognition by proposing an alignment method that uses pseudo-labels from a fine-tuned HuBERT model, achieving better performance than state-of-the-art methods on the IEMOCAP dataset.
Speech emotion recognition (SER) systems aim to recognize human emotional state during human-computer interaction. Most existing SER systems are trained based on utterance-level labels. However, not all frames in an audio have affective states consistent with utterance-level label, which makes it difficult for the model to distinguish the true emotion of the audio and perform poorly. To address this problem, we propose a frame-level emotional state alignment method for SER. First, we fine-tune HuBERT model to obtain a SER system with task-adaptive pretraining (TAPT) method, and extract embeddings from its transformer layers to form frame-level pseudo-emotion labels with clustering. Then, the pseudo labels are used to pretrain HuBERT. Hence, the each frame output of HuBERT has corresponding emotional information. Finally, we fine-tune the above pretrained HuBERT for SER by adding an attention layer on the top of it, which can focus only on those frames that are emotionally more consistent with utterance-level label. The experimental results performed on IEMOCAP indicate that our proposed method performs better than state-of-the-art (SOTA) methods.