Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning
This work addresses audio captioning for applications like accessibility or content analysis, presenting an incremental improvement over existing retrieval-augmented methods.
The paper tackles the problem of improving audio captioning by proposing Generation-Assisted Multimodal Querying, which generates text descriptions from input audio to enable multimodal retrieval from a knowledge base, achieving state-of-the-art results on AudioCaps, Clotho, and Auto-ACD benchmarks.
Retrieval-augmented generation can improve audio captioning by incorporating relevant audio-text pairs from a knowledge base. Existing methods typically rely solely on the input audio as a unimodal retrieval query. In contrast, we propose Generation-Assisted Multimodal Querying, which generates a text description of the input audio to enable multimodal querying. This approach aligns the query modality with the audio-text structure of the knowledge base, leading to more effective retrieval. Furthermore, we introduce a novel progressive learning strategy that gradually increases the number of interleaved audio-text pairs to enhance the training process. Our experiments on AudioCaps, Clotho, and Auto-ACD demonstrate that our approach achieves state-of-the-art results across these benchmarks.