Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions
This work addresses the challenge of improving retrieval accuracy in text-video systems, which is important for applications like video search and recommendation, though it appears to be an incremental improvement over existing caption-based methods.
The paper tackles the problem of inaccurate text-video retrieval by proposing NarVid, a framework that strategically leverages frame-level captions to capture rich video semantics and filter incorrect information, achieving state-of-the-art performance on various benchmark datasets.
In recent text-video retrieval, the use of additional captions from vision-language models has shown promising effects on the performance. However, existing models using additional captions often have struggled to capture the rich semantics, including temporal changes, inherent in the video. In addition, incorrect information caused by generative models can lead to inaccurate retrieval. To address these issues, we propose a new framework, Narrating the Video (NarVid), which strategically leverages the comprehensive information available from frame-level captions, the narration. The proposed NarVid exploits narration in multiple ways: 1) feature enhancement through cross-modal interactions between narration and video, 2) query-aware adaptive filtering to suppress irrelevant or incorrect information, 3) dual-modal matching score by adding query-video similarity and query-narration similarity, and 4) hard-negative loss to learn discriminative features from multiple perspectives using the two similarities from different views. Experimental results demonstrate that NarVid achieves state-of-the-art performance on various benchmark datasets.