Prompt2LVideos: Exploring Prompts for Understanding Long-Form Multimodal Videos
This work addresses the problem of automated comprehension of long-form videos for researchers and practitioners, but it is incremental as it focuses on dataset creation and baseline analysis without presenting new methods or significant results.
The paper tackles the challenge of understanding long-form multimodal videos in educational and news domains by introducing a new dataset of lectures and news videos, and it advocates for exploring prompt engineering techniques to address limitations in existing automated methods.
Learning multimodal video understanding typically relies on datasets comprising video clips paired with manually annotated captions. However, this becomes even more challenging when dealing with long-form videos, lasting from minutes to hours, in educational and news domains due to the need for more annotators with subject expertise. Hence, there arises a need for automated solutions. Recent advancements in Large Language Models (LLMs) promise to capture concise and informative content that allows the comprehension of entire videos by leveraging Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR) technologies. ASR provides textual content from audio, while OCR extracts textual content from specific frames. This paper introduces a dataset comprising long-form lectures and news videos. We present baseline approaches to understand their limitations on this dataset and advocate for exploring prompt engineering techniques to comprehend long-form multimodal video datasets comprehensively.