End-to-end Dense Video Captioning as Sequence Generation
This work addresses the problem of automating detailed video descriptions for applications like video indexing and accessibility, representing an incremental improvement by integrating complex tasks into large-scale pretrained models.
The paper tackles dense video captioning by modeling event detection and description generation as a single sequence generation task, achieving encouraging results on YouCook2 and ViTT datasets.
Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event. Previous approaches usually follow a two-stage generative process, which first proposes a segment for each event, then renders a caption for each identified segment. Recent advances in large-scale sequence generation pretraining have seen great success in unifying task formulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this powerful paradigm. In this work, we show how to model the two subtasks of dense video captioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks such as end-to-end dense video captioning integrated into large-scale pretrained models.