Efficient Audio Captioning Transformer with Patchout and Text Guidance
This work addresses the multi-modal translation problem of generating textual descriptions from audio clips, with incremental improvements for the audio captioning domain.
The paper tackles automated audio captioning by proposing a full Transformer architecture with Patchout and text guidance, which improves performance and reduces computational complexity, winning the Judges Award at the DCASE Challenge 2022.
Automated audio captioning is multi-modal translation task that aim to generate textual descriptions for a given audio clip. In this paper we propose a full Transformer architecture that utilizes Patchout as proposed in [1], significantly reducing the computational complexity and avoiding overfitting. The caption generation is partly conditioned on textual AudioSet tags extracted by a pre-trained classification model which is fine-tuned to maximize the semantic similarity between AudioSet labels and ground truth captions. To mitigate the data scarcity problem of Automated Audio Captioning we introduce transfer learning from an upstream audio-related task and an enlarged in-domain dataset. Moreover, we propose a method to apply Mixup augmentation for AAC. Ablation studies are carried out to investigate how Patchout and text guidance contribute to the final performance. The results show that the proposed techniques improve the performance of our system and while reducing the computational complexity. Our proposed method received the Judges Award at the Task6A of DCASE Challenge 2022.