Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning
This work addresses the cross-modal challenge in automated audio captioning for applications like accessibility or media indexing, though it is incremental as it builds on pre-trained models and contrastive learning techniques.
The paper tackles the problem of automated audio captioning by proposing CLIP-AAC, a system that learns interactive cross-modality representations using both acoustic and textual information with contrastive learning, resulting in significant performance gains over baselines on the Clotho dataset.
Automated Audio captioning (AAC) is a cross-modal task that generates natural language to describe the content of input audio. Most prior works usually extract single-modality acoustic features and are therefore sub-optimal for the cross-modal decoding task. In this work, we propose a novel AAC system called CLIP-AAC to learn interactive cross-modality representation with both acoustic and textual information. Specifically, the proposed CLIP-AAC introduces an audio-head and a text-head in the pre-trained encoder to extract audio-text information. Furthermore, we also apply contrastive learning to narrow the domain difference by learning the correspondence between the audio signal and its paired captions. Experimental results show that the proposed CLIP-AAC approach surpasses the best baseline by a significant margin on the Clotho dataset in terms of NLP evaluation metrics. The ablation study indicates that both the pre-trained model and contrastive learning contribute to the performance gain of the AAC model.