Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training
This provides a new large-scale dataset for video understanding tasks, addressing a bottleneck for researchers in vision-language AI, though it is incremental as it builds on existing data collection methods.
The authors tackled the problem of limited video-sentence datasets for vision-language pre-training by creating Auto-captions on GIF, a large-scale dataset automatically extracted from web pages, which when used to pre-train a Transformer-based model yielded compelling generalizability on the MSR-VTT benchmark.
In this work, we present Auto-captions on GIF, which is a new large-scale pre-training dataset for generic video understanding. All video-sentence pairs are created by automatically extracting and filtering video caption annotations from billions of web pages. Auto-captions on GIF dataset can be utilized to pre-train the generic feature representation or encoder-decoder structure for video captioning, and other downstream tasks (e.g., sentence localization in videos, video question answering, etc.) as well. We present a detailed analysis of Auto-captions on GIF dataset in comparison to existing video-sentence datasets. We also provide an evaluation of a Transformer-based encoder-decoder structure for vision-language pre-training, which is further adapted to video captioning downstream task and yields the compelling generalizability on MSR-VTT. The dataset is available at \url{http://www.auto-video-captions.top/2020/dataset}.