H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understanding
This work addresses the need for better assessment tools for video understanding models, particularly for researchers and developers in AI and multimodal systems, though it is incremental as it builds upon existing benchmark frameworks.
The authors tackled the limitations of existing video understanding benchmarks by proposing H2VU-Benchmark, which includes extended video durations (3 seconds to 1.5 hours), comprehensive tasks like countercommonsense comprehension, and enriched first-person streaming video data, revealing that current multimodal large language models have substantial room for improvement in these new evaluation tasks.
With the rapid development of multimodal models, the demand for assessing video understanding capabilities has been steadily increasing. However, existing benchmarks for evaluating video understanding exhibit significant limitations in coverage, task diversity, and scene adaptability. These shortcomings hinder the accurate assessment of models' comprehensive video understanding capabilities. To tackle this challenge, we propose a hierarchical and holistic video understanding (H2VU) benchmark designed to evaluate both general video and online streaming video comprehension. This benchmark contributes three key features: Extended video duration: Spanning videos from brief 3-second clips to comprehensive 1.5-hour recordings, thereby bridging the temporal gaps found in current benchmarks. Comprehensive assessment tasks: Beyond traditional perceptual and reasoning tasks, we have introduced modules for countercommonsense comprehension and trajectory state tracking. These additions test the models' deep understanding capabilities beyond mere prior knowledge. Enriched video data: To keep pace with the rapid evolution of current AI agents, we have expanded first-person streaming video datasets. This expansion allows for the exploration of multimodal models' performance in understanding streaming videos from a first-person perspective. Extensive results from H2VU reveal that existing multimodal large language models (MLLMs) possess substantial potential for improvement in our newly proposed evaluation tasks. We expect that H2VU will facilitate advancements in video understanding research by offering a comprehensive and in-depth analysis of MLLMs.