Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA
This addresses the problem of computational redundancy in long-form video QA for researchers and practitioners, offering an incremental improvement in efficiency.
The paper tackles the inefficiency of processing all frames in long-form video question answering by proposing LVNet, a modular framework with a Hierarchical Keyframe Selector that selects minimal informative frames per question, achieving state-of-the-art performance on four benchmark datasets.
Long-form videos that span across wide temporal intervals are highly information redundant and contain multiple distinct events or entities that are often loosely related. Therefore, when performing long-form video question answering (LVQA), all information necessary to generate a correct response can often be contained within a small subset of frames. Recent literature leverage large language models (LLMs) in LVQA benchmarks, achieving exceptional performance, while relying on vision language models (VLMs) to convert all visual content within videos into natural language. Such VLMs often independently caption a large number of frames uniformly sampled from long videos, which is not efficient and can mostly be redundant. Motivated by this inefficiency, we propose LVNet, a modular and training-free framework featuring a novel Hierarchical Keyframe Selector (HKS) that efficiently selects a minimal set of informative frames tailored to each question. LVNet's modularity allows easy integration with existing approaches for more efficient LVQA. We achieve state-of-the-art performance among similarly configured models across four benchmark LVQA datasets: EgoSchema, NExT-QA, IntentQA, VideoMME. The code can be found at https://github.com/jongwoopark7978/LVNet