A Modular Approach for Multimodal Summarization of TV Shows
This addresses the challenge of summarizing complex, multimodal TV content for AI applications, though it is incremental as it builds on existing methods with a modular design.
The paper tackles the problem of summarizing TV shows by introducing a modular approach that decomposes the task into specialized sub-tasks, resulting in higher quality summaries as measured by ROUGE and a new fact-based metric on the SummScreen3D dataset.
In this paper we address the task of summarizing television shows, which touches key areas in AI research: complex reasoning, multiple modalities, and long narratives. We present a modular approach where separate components perform specialized sub-tasks which we argue affords greater flexibility compared to end-to-end methods. Our modules involve detecting scene boundaries, reordering scenes so as to minimize the number of cuts between different events, converting visual information to text, summarizing the dialogue in each scene, and fusing the scene summaries into a final summary for the entire episode. We also present a new metric, PRISMA (Precision and Recall EvaluatIon of Summary FActs), to measure both precision and recall of generated summaries, which we decompose into atomic facts. Tested on the recently released SummScreen3D dataset, our method produces higher quality summaries than comparison models, as measured with ROUGE and our new fact-based metric, and as assessed by human evaluators.