Multi-Granularity and Multi-modal Feature Interaction Approach for Text Video Retrieval
This work improves text-to-video retrieval, a key task in multimedia AI, by better leveraging multi-granularity text features and audio, though it is incremental as it builds on existing alignment approaches.
The paper tackles the problem of aligning text and video representations in text-to-video retrieval by addressing issues like varying word importance and underutilized audio information, resulting in a method that outperforms state-of-the-art methods on benchmark datasets such as MSR-VTT, MSVD, and DiDeMo.
The key of the text-to-video retrieval (TVR) task lies in learning the unique similarity between each pair of text (consisting of words) and video (consisting of audio and image frames) representations. However, some problems exist in the representation alignment of video and text, such as a text, and further each word, are of different importance for video frames. Besides, audio usually carries additional or critical information for TVR in the case that frames carry little valid information. Therefore, in TVR task, multi-granularity representation of text, including whole sentence and every word, and the modal of audio are salutary which are underutilized in most existing works. To address this, we propose a novel multi-granularity feature interaction module called MGFI, consisting of text-frame and word-frame, for video-text representations alignment. Moreover, we introduce a cross-modal feature interaction module of audio and text called CMFI to solve the problem of insufficient expression of frames in the video. Experiments on benchmark datasets such as MSR-VTT, MSVD, DiDeMo show that the proposed method outperforms the existing state-of-the-art methods.