TubeDETR: Spatio-Temporal Video Grounding with Transformers
This addresses the challenging task of video grounding for applications like video understanding and retrieval, representing an incremental advancement with specific gains.
The paper tackles the problem of localizing a spatio-temporal tube in a video based on a text query, proposing TubeDETR, a transformer-based architecture that improves state-of-the-art performance on VidSTG and HC-STVG benchmarks.
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query. This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal interactions. To address this task, we propose TubeDETR, a transformer-based architecture inspired by the recent success of such models for text-conditioned object detection. Our model notably includes: (i) an efficient video and text encoder that models spatial multi-modal interactions over sparsely sampled frames and (ii) a space-time decoder that jointly performs spatio-temporal localization. We demonstrate the advantage of our proposed components through an extensive ablation study. We also evaluate our full approach on the spatio-temporal video grounding task and demonstrate improvements over the state of the art on the challenging VidSTG and HC-STVG benchmarks. Code and trained models are publicly available at https://antoyang.github.io/tubedetr.html.