Zero-shot Natural Language Video Localization
This addresses the high annotation costs in video understanding for researchers and practitioners, offering a zero-shot approach that is incremental but practical.
The paper tackles the problem of localizing moments in videos using natural language queries without requiring annotated video-language pairs, by generating pseudo-supervision from unpaired data and training a model that outperforms baselines and some supervised methods on Charades-STA and ActivityNet-Captions.
Understanding videos to localize moments with natural language often requires large expensive annotated video regions paired with language queries. To eliminate the annotation costs, we make a first attempt to train a natural language video localization model in zero-shot manner. Inspired by unsupervised image captioning setup, we merely require random text corpora, unlabeled video collections, and an off-the-shelf object detector to train a model. With the unpaired data, we propose to generate pseudo-supervision of candidate temporal regions and corresponding query sentences, and develop a simple NLVL model to train with the pseudo-supervision. Our empirical validations show that the proposed pseudo-supervised method outperforms several baseline approaches and a number of methods using stronger supervision on Charades-STA and ActivityNet-Captions.