Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs
This work addresses the problem of costly and skill-agnostic evaluation for video MLLMs, offering a scalable tool for researchers and developers, though it is incremental as it builds on existing benchmark methods.
The paper tackles the inefficiency of existing video benchmarks for evaluating multimodal large language models (MLLMs) by proposing VideoNIAH, a synthetic framework that generates videos with inserted visual 'needles' and automated query-response pairs, resulting in the VNBench benchmark that reveals significant differences in model capabilities across tasks like retrieval and ordering.
Video understanding is a crucial next step for multimodal large language models (MLLMs). Various benchmarks are introduced for better evaluating the MLLMs. Nevertheless, current video benchmarks are still inefficient for evaluating video models during iterative development due to the high cost of constructing datasets and the difficulty in isolating specific skills. In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. VideoNIAH decouples video content from their query-responses by inserting unrelated visual 'needles' into original videos. The framework automates the generation of query-response pairs using predefined rules, minimizing manual labor. The queries focus on specific aspects of video understanding, enabling more skill-specific evaluations. The separation between video content and the queries also allow for increased video variety and evaluations across different lengths. Utilizing VideoNIAH, we compile a video benchmark VNBench, which includes tasks such as retrieval, ordering, and counting to evaluate three key aspects of video understanding: temporal perception, chronological ordering, and spatio-temporal coherence. We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities across various tasks. Additionally, we perform an in-depth analysis of the test results and model configurations. Based on these findings, we provide some advice for improving video MLLM training, offering valuable insights to guide future research and model development. The code and data are available at https://github.com/joez17/VideoNIAH.