Learning Summary-Worthy Visual Representation for Abstractive Summarization in Video
This work improves multimodal abstractive summarization for videos, which is important for applications like content indexing and accessibility, though it is incremental as it builds on existing generative pre-trained language models.
The paper tackles the problem of generating abstractive summaries for videos by addressing the limitation of general-purpose visual features that overlook summary-worthy information, resulting in a method that outperforms baselines on three public datasets and shows significant improvement on small or limited datasets.
Multimodal abstractive summarization for videos (MAS) requires generating a concise textual summary to describe the highlights of a video according to multimodal resources, in our case, the video content and its transcript. Inspired by the success of the large-scale generative pre-trained language model (GPLM) in generating high-quality textual content (e.g., summary), recent MAS methods have proposed to adapt the GPLM to this task by equipping it with the visual information, which is often obtained through a general-purpose visual feature extractor. However, the generally extracted visual features may overlook some summary-worthy visual information, which impedes model performance. In this work, we propose a novel approach to learning the summary-worthy visual representation that facilitates abstractive summarization. Our method exploits the summary-worthy information from both the cross-modal transcript data and the knowledge that distills from the pseudo summary. Extensive experiments on three public multimodal datasets show that our method outperforms all competing baselines. Furthermore, with the advantages of summary-worthy visual information, our model can have a significant improvement on small datasets or even datasets with limited training data.