Streamlined Dense Video Captioning
This addresses the challenge of generating accurate and coherent descriptions for videos, which is important for applications like video summarization and accessibility, but it is incremental as it builds on existing proposal-based methods.
The paper tackled the problem of redundant and inconsistent sentences in dense video captioning by proposing a framework that models temporal dependency across events and leverages context for coherent storytelling, achieving outstanding performances on the ActivityNet Captions dataset.
Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events. Most existing approaches handle this problem by first detecting event proposals from a video and then captioning on a subset of the proposals. As a result, the generated sentences are prone to be redundant or inconsistent since they fail to consider temporal dependency between events. To tackle this challenge, we propose a novel dense video captioning framework, which models temporal dependency across events in a video explicitly and leverages visual and linguistic context from prior events for coherent storytelling. This objective is achieved by 1) integrating an event sequence generation network to select a sequence of event proposals adaptively, and 2) feeding the sequence of event proposals to our sequential video captioning network, which is trained by reinforcement learning with two-level rewards at both event and episode levels for better context modeling. The proposed technique achieves outstanding performances on ActivityNet Captions dataset in most metrics.