Audio Captioning Using Sound Event Detection
This is an incremental improvement for audio captioning systems, potentially benefiting applications in accessibility or media analysis.
The paper tackles audio captioning by integrating sound event detection with an encoder-decoder model using pretrained audio features, achieving significantly better results than the baseline on the Clotho dataset across all evaluation metrics.
This technical report proposes an audio captioning system for DCASE 2021 Task 6 audio captioning challenge. Our proposed model is based on an encoder-decoder architecture with bi-directional Gated Recurrent Units (BiGRU) using pretrained audio features and sound event detection. A pretrained neural network (PANN) is used to extract audio features and Word2Vec is selected with the aim of extracting word embeddings from the audio captions. To create semantically meaningful captions, we extract sound events from the audio clips and feed the encoder-decoder architecture with sound events in addition to PANNs features. Our experiments on the Clotho dataset show that our proposed method significantly achieves better results than the challenge baseline model across all evaluation metrics.