Implicit and Explicit Commonsense for Multi-sentence Video Captioning
This work addresses the challenge of producing coherent and causally-aware video descriptions for applications like robotics and video analysis, though it builds incrementally on existing captioning approaches.
The paper tackles the problem of generating multi-sentence video captions by addressing the lack of commonsense knowledge in existing methods, proposing a Transformer-based model that incorporates implicit and explicit commonsense to enhance caption quality, achieving up to 57% improvement in METEOR and 8.5% in CIDEr on instruction generation and state-of-the-art results on ActivityNet Captions.
Existing dense or paragraph video captioning approaches rely on holistic representations of videos, possibly coupled with learned object/action representations, to condition hierarchical language decoders. However, they fundamentally lack the commonsense knowledge of the world required to reason about progression of events, causality, and even the function of certain objects within a scene. To address this limitation we propose a novel video captioning Transformer-based model, that takes into account both implicit (visuo-lingual and purely linguistic) and explicit (knowledge-base) commonsense knowledge. We show that these forms of knowledge, in isolation and in combination, enhance the quality of produced captions. Further, inspired by imitation learning, we propose a new task of instruction generation, where the goal is to produce a set of linguistic instructions from a video demonstration of its performance. We formalize the task using the ALFRED dataset [54] generated using an AI2-THOR environment. While instruction generation is conceptually similar to paragraph captioning, it differs in the fact that it exhibits stronger object persistence, as well as spatially-aware and causal sentence structure. We show that our commonsense knowledge enhanced approach produces significant improvements on this task (up to 57% in METEOR and 8.5% in CIDEr), as well as the state-of-the-art result on more traditional video captioning in the ActivityNet Captions dataset [29].