Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval
This addresses the challenge of reducing annotation costs for VCMR in narrative videos, though it is incremental as it builds on prior unsupervised methods.
The paper tackles the problem of video corpus moment retrieval (VCMR) by proposing a self-supervised learning framework called MPGN to generate pseudo queries from multimodal data, eliminating the need for expensive query annotations, and achieves competitive results on the TVR dataset.
Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural language query. For narrative videos, e.g., dramas or movies, the holistic understanding of temporal dynamics and multimodal reasoning is crucial. Previous works have shown promising results; however, they relied on the expensive query annotations for VCMR, i.e., the corresponding moment intervals. To overcome this problem, we propose a self-supervised learning framework: Modal-specific Pseudo Query Generation Network (MPGN). First, MPGN selects candidate temporal moments via subtitle-based moment sampling. Then, it generates pseudo queries exploiting both visual and textual information from the selected temporal moments. Through the multimodal information in the pseudo queries, we show that MPGN successfully learns to localize the video corpus moment without any explicit annotation. We validate the effectiveness of MPGN on the TVR dataset, showing competitive results compared with both supervised models and unsupervised setting models.