Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition
This work addresses speech emotion recognition, a domain-specific task, by integrating multiple existing models for data augmentation, which is incremental in nature.
The paper tackled speech emotion recognition by combining a speech pre-trained model (data2vec), GPT-4 for text generation, and Azure TTS for speech synthesis to create synthetic emotional data, achieving improved performance on the IEMOCAP dataset compared to other augmentation methods.
In this paper, we explored how to boost speech emotion recognition (SER) with the state-of-the-art speech pre-trained model (PTM), data2vec, text generation technique, GPT-4, and speech synthesis technique, Azure TTS. First, we investigated the representation ability of different speech self-supervised pre-trained models, and we found that data2vec has a good representation ability on the SER task. Second, we employed a powerful large language model (LLM), GPT-4, and emotional text-to-speech (TTS) model, Azure TTS, to generate emotionally congruent text and speech. We carefully designed the text prompt and dataset construction, to obtain the synthetic emotional speech data with high quality. Third, we studied different ways of data augmentation to promote the SER task with synthetic speech, including random mixing, adversarial training, transfer learning, and curriculum learning. Experiments and ablation studies on the IEMOCAP dataset demonstrate the effectiveness of our method, compared with other data augmentation methods, and data augmentation with other synthetic data.