Unifying Event Detection and Captioning as Sequence Generation via Pre-Training
This work addresses the problem of dense video captioning for video understanding applications, offering a novel unified approach that improves event detection and captioning performance.
The paper tackles dense video captioning by unifying event detection and captioning into a sequence generation task, using a pre-training and fine-tuning framework to enhance inter-task association and exploit temporal dependencies between events. Experiments on ActivityNet show the model outperforms state-of-the-art methods, with further improvements from pre-training on large-scale video-text data.
Dense video captioning aims to generate corresponding text descriptions for a series of events in the untrimmed video, which can be divided into two sub-tasks, event detection and event captioning. Unlike previous works that tackle the two sub-tasks separately, recent works have focused on enhancing the inter-task association between the two sub-tasks. However, designing inter-task interactions for event detection and captioning is not trivial due to the large differences in their task specific solutions. Besides, previous event detection methods normally ignore temporal dependencies between events, leading to event redundancy or inconsistency problems. To tackle above the two defects, in this paper, we define event detection as a sequence generation task and propose a unified pre-training and fine-tuning framework to naturally enhance the inter-task association between event detection and captioning. Since the model predicts each event with previous events as context, the inter-dependency between events is fully exploited and thus our model can detect more diverse and consistent events in the video. Experiments on the ActivityNet dataset show that our model outperforms the state-of-the-art methods, and can be further boosted when pre-trained on extra large-scale video-text data. Code is available at \url{https://github.com/QiQAng/UEDVC}.