EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation
This work addresses the limitation of existing audio captioning methods that ignore emotional context, offering a novel dataset and approach for researchers in audio-language modeling, though it is incremental in leveraging synthetic data generation.
The authors tackled the problem of audio captioning by generating synthetic data enriched with emotion information, resulting in improved caption quality that matches the emotional tone of audio recordings, as validated through objective and subjective evaluations.
Recent progress in audio-language modeling, such as automated audio captioning, has benefited from training on synthetic data generated with the aid of large-language models. However, such approaches for environmental sound captioning have primarily focused on audio event tags and have not explored leveraging emotional information that may be present in recordings. In this work, we explore the benefit of generating emotion-augmented synthetic audio caption data by instructing ChatGPT with additional acoustic information in the form of estimated soundscape emotion. To do so, we introduce EmotionCaps, an audio captioning dataset comprised of approximately 120,000 audio clips with paired synthetic descriptions enriched with soundscape emotion recognition (SER) information. We hypothesize that this additional information will result in higher-quality captions that match the emotional tone of the audio recording, which will, in turn, improve the performance of captioning models trained with this data. We test this hypothesis through both objective and subjective evaluation, comparing models trained with the EmotionCaps dataset to multiple baseline models. Our findings challenge current approaches to captioning and suggest new directions for developing and assessing captioning models.