SummDiff: Generative Modeling of Video Summarization with Diffusion
This addresses the challenge of generating multiple plausible video summaries that reflect varying human perspectives, which is an incremental improvement over deterministic methods.
The paper tackled the problem of video summarization by framing it as a conditional generation task to capture the inherent subjectivity of good summaries, resulting in a method that achieves state-of-the-art performance on various benchmarks and produces summaries aligning closely with individual annotator preferences.
Video summarization is a task of shortening a video by choosing a subset of frames while preserving its essential moments. Despite the innate subjectivity of the task, previous works have deterministically regressed to an averaged frame score over multiple raters, ignoring the inherent subjectivity of what constitutes a good summary. We propose a novel problem formulation by framing video summarization as a conditional generation task, allowing a model to learn the distribution of good summaries and to generate multiple plausible summaries that better reflect varying human perspectives. Adopting diffusion models for the first time in video summarization, our proposed method, SummDiff, dynamically adapts to visual contexts and generates multiple candidate summaries conditioned on the input video. Extensive experiments demonstrate that SummDiff not only achieves the state-of-the-art performance on various benchmarks but also produces summaries that closely align with individual annotator preferences. Moreover, we provide a deeper insight with novel metrics from an analysis of the knapsack, which is an important last step of generating summaries but has been overlooked in evaluation.