QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding
For video understanding tasks using Video-LLMs, QCA provides an efficient and effective method to select informative frames without additional training, improving both performance and computational efficiency.
The paper proposes QCA, a training-free keyframe selection framework that reduces temporal redundancy in long videos by jointly modeling query relevance and content deviation. It achieves state-of-the-art performance, e.g., 67.8% on LongVideoBench with 128 frames, outperforming GPT-4o's 66.7% with 256 frames.
Video understanding is often plagued by severe temporal redundancy, where processing dense frame sequences is both semantically inefficient and computationally expensive. This challenge is further amplified when only a small subset of frames is truly relevant to the given query. In this paper, we propose a Query- and Content-Aware (QCA) keyframe selection framework that can select a compact yet information-rich set of frames from long videos. QCA first partitions the video into temporal segments and estimates the information contribution of each segment by jointly modeling query relevance and content deviation, and dynamically allocates keyframe budget to each segment. Within each segment, QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query. Crucially, our method requires no additional training and can be seamlessly integrated into existing Video-LLMs. Extensive experiments across multiple long video understanding benchmarks demonstrate that our proposed approach achieves state-of-the-art performance and has strong generalization ability. For instance, QCA achieves 67.8\% on LongVideoBench using 128 frames, while GPT-4o achieves 66.7\% using 256 frames. Our codes are available in \href{https://github.com/hktk07/QCA}{GitHub}.