Multi-Modal interpretable automatic video captioning
This work addresses the need for more accurate and interpretable video captions for applications in multimedia analysis and accessibility, though it is incremental in improving multi-modal integration.
The paper tackles the problem of video captioning by integrating audio information with visual cues, which are often neglected, and achieves favorable performance against state-of-the-art models on MSR-VTT and VATEX benchmarks.
Video captioning aims to describe video contents using natural language format that involves understanding and interpreting scenes, actions and events that occurs simultaneously on the view. Current approaches have mainly concentrated on visual cues, often neglecting the rich information available from other important modality of audio information, including their inter-dependencies. In this work, we introduce a novel video captioning method trained with multi-modal contrastive loss that emphasizes both multi-modal integration and interpretability. Our approach is designed to capture the dependency between these modalities, resulting in more accurate, thus pertinent captions. Furthermore, we highlight the importance of interpretability, employing multiple attention mechanisms that provide explanation into the model's decision-making process. Our experimental results demonstrate that our proposed method performs favorably against the state-of the-art models on commonly used benchmark datasets of MSR-VTT and VATEX.