Do Video Language Models Really Know Where to Look? Diagnosing Attention Failures in Video Language Models
This addresses a bottleneck in efficient video understanding for AI researchers, but it is incremental as it diagnoses an existing issue without proposing a new solution.
The paper tackles the problem of whether vision-language encoders in video multimodal large language models can identify the most informative frames, finding that popular encoders critically fail to guide attention appropriately, suggesting better keyframe identification techniques are needed.
Recent advances in multimodal large language models (MLLMs) have led to much progress in video understanding tasks. To avoid the heavy computational cost of processing all frames, these models typically rely on keyframe sampling methods guided by vision-language encoders (\textit{e.g.,} SigLIP). However, it remains unclear whether such encoders can truly identify the most informative frames. In this work, we provide several empirical pieces of evidence revealing that popular vision encoders critically suffer from their limited capability to identify where the MLLM should look inside the video to handle the given textual query appropriately. Our findings suggest that the development of better keyframe identification techniques may be necessary for efficient video MLLMs.