O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
This work addresses video captioning for applications like accessibility or content indexing, but it is incremental as it builds on existing non-autoregressive and object-focused approaches.
The paper tackles the problem of generating captions for videos by focusing on key objects, proposing O2NA, a non-autoregressive method that identifies and positions focused objects first, then refines the caption, achieving competitive results on MSR-VTT and MSVD datasets with higher diversity and inference speed.
Video captioning combines video understanding and language generation. Different from image captioning that describes a static image with details of almost every object, video captioning usually considers a sequence of frames and biases towards focused objects, e.g., the objects that stay in focus regardless of the changing background. Therefore, detecting and properly accommodating focused objects is critical in video captioning. To enforce the description of focused objects and achieve controllable video captioning, we propose an Object-Oriented Non-Autoregressive approach (O2NA), which performs caption generation in three steps: 1) identify the focused objects and predict their locations in the target caption; 2) generate the related attribute words and relation words of these focused objects to form a draft caption; and 3) combine video information to refine the draft caption to a fluent final caption. Since the focused objects are generated and located ahead of other words, it is difficult to apply the word-by-word autoregressive generation process; instead, we adopt a non-autoregressive approach. The experiments on two benchmark datasets, i.e., MSR-VTT and MSVD, demonstrate the effectiveness of O2NA, which achieves results competitive with the state-of-the-arts but with both higher diversity and higher inference speed.