Progressive Localization Networks for Language-based Moment Localization
This work addresses a domain-specific problem in video understanding for applications like video retrieval, but it is incremental as it builds on existing candidate sampling methods with a coarse-to-fine refinement approach.
The paper tackles the challenge of language-based video moment localization, where varying moment lengths make fixed-granularity candidate sampling suboptimal, and proposes a multi-stage Progressive Localization Network (PLN) that achieves improved performance, particularly for short moments in long videos, as demonstrated on three public datasets.
This paper targets the task of language-based video moment localization. The language-based setting of this task allows for an open set of target activities, resulting in a large variation of the temporal lengths of video moments. Most existing methods prefer to first sample sufficient candidate moments with various temporal lengths, and then match them with the given query to determine the target moment. However, candidate moments generated with a fixed temporal granularity may be suboptimal to handle the large variation in moment lengths. To this end, we propose a novel multi-stage Progressive Localization Network (PLN) which progressively localizes the target moment in a coarse-to-fine manner. Specifically, each stage of PLN has a localization branch, and focuses on candidate moments that are generated with a specific temporal granularity. The temporal granularities of candidate moments are different across the stages. Moreover, we devise a conditional feature manipulation module and an upsampling connection to bridge the multiple localization branches. In this fashion, the later stages are able to absorb the previously learned information, thus facilitating the more fine-grained localization. Extensive experiments on three public datasets demonstrate the effectiveness of our proposed PLN for language-based moment localization, especially for localizing short moments in long videos.