Syntax Customized Video Captioning by Imitating Exemplar Sentences
This addresses the need for varied sentence structures in video captioning, offering a novel approach to enhance descriptive diversity, though it is incremental in applying syntax imitation to an existing domain.
The paper tackles the problem of generating diverse video captions by customizing them to imitate the syntax of given exemplar sentences, introducing the Syntax Customized Video Captioning (SCVC) task and a model that achieves this through a hierarchical syntax encoder and conditioned decoder, with experiments showing improved diversity and coherence.
Enhancing the diversity of sentences to describe video contents is an important problem arising in recent video captioning research. In this paper, we explore this problem from a novel perspective of customizing video captions by imitating exemplar sentence syntaxes. Specifically, given a video and any syntax-valid exemplar sentence, we introduce a new task of Syntax Customized Video Captioning (SCVC) aiming to generate one caption which not only semantically describes the video contents but also syntactically imitates the given exemplar sentence. To tackle the SCVC task, we propose a novel video captioning model, where a hierarchical sentence syntax encoder is firstly designed to extract the syntactic structure of the exemplar sentence, then a syntax conditioned caption decoder is devised to generate the syntactically structured caption expressing video semantics. As there is no available syntax customized groundtruth video captions, we tackle such a challenge by proposing a new training strategy, which leverages the traditional pairwise video captioning data and our collected exemplar sentences to accomplish the model learning. Extensive experiments, in terms of semantic, syntactic, fluency, and diversity evaluations, clearly demonstrate our model capability to generate syntax-varied and semantics-coherent video captions that well imitate different exemplar sentences with enriched diversities.