HourVideo: 1-Hour Video-Language Understanding
This addresses the problem of evaluating multimodal AI capabilities on long videos for researchers, but it is incremental as it builds on existing datasets like Ego4D.
The authors introduced HourVideo, a benchmark dataset for hour-long video-language understanding, and found that multimodal models like GPT-4 and LLaVA-NeXT perform only slightly better than random chance, while human experts significantly outperform the best model (85.0% vs. 37.3%).
We present HourVideo, a benchmark dataset for hour-long video-language understanding. Our dataset consists of a novel task suite comprising summarization, perception (recall, tracking), visual reasoning (spatial, temporal, predictive, causal, counterfactual), and navigation (room-to-room, object retrieval) tasks. HourVideo includes 500 manually curated egocentric videos from the Ego4D dataset, spanning durations of 20 to 120 minutes, and features 12,976 high-quality, five-way multiple-choice questions. Benchmarking results reveal that multimodal models, including GPT-4 and LLaVA-NeXT, achieve marginal improvements over random chance. In stark contrast, human experts significantly outperform the state-of-the-art long-context multimodal model, Gemini Pro 1.5 (85.0% vs. 37.3%), highlighting a substantial gap in multimodal capabilities. Our benchmark, evaluation toolkit, prompts, and documentation are available at https://hourvideo.stanford.edu