Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations
This work addresses the challenge of limited multimodal data and alignment issues in emotion recognition for human-machine interaction systems, offering an incremental improvement over existing methods.
The paper tackles the problem of emotion recognition in conversations by proposing the Qieemo framework, which uses a pretrained ASR model to extract aligned textual and emotional features from audio, achieving absolute improvements of 3.0%, 1.2%, and 1.9% over benchmark models on the IEMOCAP dataset.
Emotion recognition plays a pivotal role in intelligent human-machine interaction systems. Multimodal approaches benefit from the fusion of diverse modalities, thereby improving the recognition accuracy. However, the lack of high-quality multimodal data and the challenge of achieving optimal alignment between different modalities significantly limit the potential for improvement in multimodal approaches. In this paper, the proposed Qieemo framework effectively utilizes the pretrained automatic speech recognition (ASR) model backbone which contains naturally frame aligned textual and emotional features, to achieve precise emotion classification solely based on the audio modality. Furthermore, we design the multimodal fusion (MMF) module and cross-modal attention (CMA) module in order to fuse the phonetic posteriorgram (PPG) and emotional features extracted by the ASR encoder for improving recognition accuracy. The experimental results on the IEMOCAP dataset demonstrate that Qieemo outperforms the benchmark unimodal, multimodal, and self-supervised models with absolute improvements of 3.0%, 1.2%, and 1.9% respectively.