Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure
This work addresses the challenge of summarizing text-heavy multimodal documents for users needing efficient content extraction, but it is incremental as it focuses on optimizing existing VLM methods rather than introducing new paradigms.
The study tackled the problem of automatically summarizing multimodal presentations using Vision-Language Models, finding that structured representations combining interleaved slides and transcripts yield the best performance, with slides extracted from videos being more cost-effective than raw video input.
Vision-Language Models (VLMs) can process visual and textual information in multiple formats: texts, images, interleaved texts and images, or even hour-long videos. In this work, we conduct fine-grained quantitative and qualitative analyses of automatic summarization of multimodal presentations using VLMs with various representations as input. From these experiments, we suggest cost-effective strategies for generating summaries from text-heavy multimodal documents under different input-length budgets using VLMs. We show that slides extracted from the video stream can be beneficially used as input against the raw video, and that a structured representation from interleaved slides and transcript provides the best performance. Finally, we reflect and comment on the nature of cross-modal interactions in multimodal presentations and share suggestions to improve the capabilities of VLMs to understand documents of this nature.