Look Closer to Ground Better: Weakly-Supervised Temporal Grounding of Sentence in Video
This addresses the problem of video understanding for applications like search and summarization, but it is incremental as it builds on existing weakly-supervised methods.
The paper tackles weakly-supervised temporal grounding of sentences in videos by localizing temporal segments without temporal annotations during training, achieving compelling performance on ActivityNet Captions and Charades-STA datasets.
In this paper, we study the problem of weakly-supervised temporal grounding of sentence in video. Specifically, given an untrimmed video and a query sentence, our goal is to localize a temporal segment in the video that semantically corresponds to the query sentence, with no reliance on any temporal annotation during training. We propose a two-stage model to tackle this problem in a coarse-to-fine manner. In the coarse stage, we first generate a set of fixed-length temporal proposals using multi-scale sliding windows, and match their visual features against the sentence features to identify the best-matched proposal as a coarse grounding result. In the fine stage, we perform a fine-grained matching between the visual features of the frames in the best-matched proposal and the sentence features to locate the precise frame boundary of the fine grounding result. Comprehensive experiments on the ActivityNet Captions dataset and the Charades-STA dataset demonstrate that our two-stage model achieves compelling performance.