Weakly-supervised Automated Audio Captioning via text only training
This reduces the need for labor-intensive paired data collection in audio captioning, though it is incremental as it builds on existing CLAP models.
The paper tackles the problem of collecting paired audio-caption data for Automated Audio Captioning by proposing a weakly-supervised method that uses only text data and a pre-trained CLAP model, achieving up to ~83% relative performance compared to fully supervised approaches on Clotho and AudioCaps datasets.
In recent years, datasets of paired audio and captions have enabled remarkable success in automatically generating descriptions for audio clips, namely Automated Audio Captioning (AAC). However, it is labor-intensive and time-consuming to collect a sufficient number of paired audio and captions. Motivated by the recent advances in Contrastive Language-Audio Pretraining (CLAP), we propose a weakly-supervised approach to train an AAC model assuming only text data and a pre-trained CLAP model, alleviating the need for paired target data. Our approach leverages the similarity between audio and text embeddings in CLAP. During training, we learn to reconstruct the text from the CLAP text embedding, and during inference, we decode using the audio embeddings. To mitigate the modality gap between the audio and text embeddings we employ strategies to bridge the gap during training and inference stages. We evaluate our proposed method on Clotho and AudioCaps datasets demonstrating its ability to achieve a relative performance of up to ~$83\%$ compared to fully supervised approaches trained with paired target data.