A Paradigm Shift: Fully End-to-End Training for Temporal Sentence Grounding in Videos
This addresses a key bottleneck in video understanding for researchers and practitioners by improving localization accuracy, though it is incremental in optimizing existing architectures.
The paper tackles the task discrepancy issue in temporal sentence grounding in videos by proposing a fully end-to-end training paradigm that jointly optimizes the video backbone and localization head, outperforming state-of-the-art methods on two benchmarks.
Temporal sentence grounding in videos (TSGV) aims to localize a temporal segment that semantically corresponds to a sentence query from an untrimmed video. Most current methods adopt pre-trained query-agnostic visual encoders for offline feature extraction, and the video backbones are frozen and not optimized for TSGV. This leads to a task discrepancy issue for the video backbone trained for visual classification, but utilized for TSGV. To bridge this gap, we propose a fully end-to-end paradigm that jointly optimizes the video backbone and localization head. We first conduct an empirical study validating the effectiveness of end-to-end learning over frozen baselines across different model scales. Furthermore, we introduce a Sentence Conditioned Adapter (SCADA), which leverages sentence features to train a small portion of video backbone parameters adaptively. SCADA facilitates the deployment of deeper network backbones with reduced memory and significantly enhances visual representation by modulating feature maps through precise integration of linguistic embeddings. Experiments on two benchmarks show that our method outperforms state-of-the-art approaches. The code and models will be released.