Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning
This work addresses the challenge of generating detailed captions for video events without full supervision, benefiting video analysis and multi-modal AI applications, though it is incremental as it builds on existing multi-modal learning approaches.
The paper tackles the problem of weakly-supervised dense event captioning in videos by leveraging audio signals, showing that audio alone nearly matches state-of-the-art visual models and combining audio with video improves performance beyond existing methods, with experiments on ActivityNet Captions dataset demonstrating these gains.
Multi-modal learning, particularly among imaging and linguistic modalities, has made amazing strides in many high-level fundamental visual understanding problems, ranging from language grounding to dense event captioning. However, much of the research has been limited to approaches that either do not take audio corresponding to video into account at all, or those that model the audio-visual correlations in service of sound or sound source localization. In this paper, we present the evidence, that audio signals can carry surprising amount of information when it comes to high-level visual-lingual tasks. Specifically, we focus on the problem of weakly-supervised dense event captioning in videos and show that audio on its own can nearly rival performance of a state-of-the-art visual model and, combined with video, can improve on the state-of-the-art performance. Extensive experiments on the ActivityNet Captions dataset show that our proposed multi-modal approach outperforms state-of-the-art unimodal methods, as well as validate specific feature representation and architecture design choices.