VVS: Video-to-Video Retrieval with Irrelevant Frame Suppression
This work addresses efficiency and accuracy trade-offs in large-scale video retrieval for applications like media search, though it is incremental as it builds on existing video-level feature methods.
The paper tackles the challenge of embedding untrimmed videos into single features for efficient content-based video retrieval by suppressing irrelevant frames, achieving state-of-the-art accuracy in video-level approaches with fast inference time close to frame-level methods.
In content-based video retrieval (CBVR), dealing with large-scale collections, efficiency is as important as accuracy; thus, several video-level feature-based studies have actively been conducted. Nevertheless, owing to the severe difficulty of embedding a lengthy and untrimmed video into a single feature, these studies have been insufficient for accurate retrieval compared to frame-level feature-based studies. In this paper, we show that appropriate suppression of irrelevant frames can provide insight into the current obstacles of the video-level approaches. Furthermore, we propose a Video-to-Video Suppression network (VVS) as a solution. VVS is an end-to-end framework that consists of an easy distractor elimination stage to identify which frames to remove and a suppression weight generation stage to determine the extent to suppress the remaining frames. This structure is intended to effectively describe an untrimmed video with varying content and meaningless information. Its efficacy is proved via extensive experiments, and we show that our approach is not only state-of-the-art in video-level approaches but also has a fast inference time despite possessing retrieval capabilities close to those of frame-level approaches. Code is available at https://github.com/sejong-rcv/VVS