Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
This work addresses the problem of ambiguous sound recognition in audio captioning for applications like multimedia analysis, though it is incremental as it builds on existing multimodal methods.
The paper tackles the challenge of recognizing ambiguous sounds in audio captioning by incorporating visual information to improve descriptions, achieving state-of-the-art results on the AudioCaps dataset.
Audio captioning aims to generate text descriptions of audio clips. In the real world, many objects produce similar sounds. How to accurately recognize ambiguous sounds is a major challenge for audio captioning. In this work, inspired by inherent human multimodal perception, we propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Specifically, we introduce an off-the-shelf visual encoder to extract video features and incorporate the visual features into an audio captioning system. Furthermore, to better exploit complementary audio-visual contexts, we propose an audio-visual attention mechanism that adaptively integrates audio and visual context and removes the redundant information in the latent space. Experimental results on AudioCaps, the largest audio captioning dataset, show that our proposed method achieves state-of-the-art results on machine translation metrics.