Datasets and Recipes for Video Temporal Grounding via Reinforcement Learning
This work addresses the limited temporal awareness and poor generalization in VTG models, offering improved accuracy and robustness for applications in video analysis and retrieval.
The paper tackles the problem of video temporal grounding (VTG) by introducing a two-stage training framework combining supervised fine-tuning with reinforcement learning, which consistently outperforms existing models on multiple benchmarks, especially in challenging scenarios.
Video Temporal Grounding (VTG) aims to localize relevant temporal segments in videos given natural language queries. Despite recent progress with large vision-language models (LVLMs) and instruction-tuning, existing approaches often suffer from limited temporal awareness and poor generalization. In this work, we introduce a two-stage training framework that integrates supervised fine-tuning with reinforcement learning (RL) to improve both the accuracy and robustness of VTG models. Our approach first leverages high-quality curated cold start data for SFT initialization, followed by difficulty-controlled RL to further enhance temporal localization and reasoning abilities. Comprehensive experiments on multiple VTG benchmarks demonstrate that our method consistently outperforms existing models, particularly in challenging and open-domain scenarios. We conduct an in-depth analysis of training strategies and dataset curation, highlighting the importance of both high-quality cold start data and difficulty-controlled RL. To facilitate further research and industrial adoption, we release all intermediate datasets, models, and code to the community.