Learning Question-Aware Keyframe Selection with Synthetic Supervision for Video Question Answering
This work addresses efficiency and reasoning challenges in video question answering for AI systems, though it is incremental as it builds on existing keyframe selection methods.
The paper tackled the problem of inefficient and diluted reasoning in video question answering by introducing a question-aware keyframe selection framework, which improved accuracy on NExT-QA, particularly for temporal and causal questions.
Large multimodal models (LMMs) have recently demonstrated remarkable performance in video question answering (VideoQA), yet reasoning over video remains challenging due to high inference cost and diluted information. Keyframe selection offers efficiency and sharper reasoning but suffers from sparse supervision and redundant frame choices when relying only on image-text similarity. We present a question-aware keyframe selection framework with two components: pseudo keyframe labels derived from LMMs that provide informative supervision and a coverage regularization that promotes diverse, complementary evidence across time. Experiments on NExT-QA show that our method significantly improves accuracy, especially for temporal and causal question types, establishing keyframe selection as an effective and learnable module for VideoQA.