VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment
This addresses the problem of evaluating compositional reasoning in video-language models for researchers, though it is incremental as it builds on existing benchmarks.
The authors introduced VELOCITI, a benchmark to assess video-language models on compositional reasoning in short videos, finding that top models like LLaVA-OneVision (44.5%) and Gemini-1.5-Pro (49.3%) perform far below human accuracy (93.0%).
A fundamental aspect of compositional reasoning in a video is associating people and their actions across time. Recent years have seen great progress in general-purpose vision or video models and a move towards long-video understanding. While exciting, we take a step back and ask: are current models good at compositional reasoning on short videos? To this end, we introduce VELOCITI, a benchmark to study Video-LLMs by disentangling and assessing the comprehension of agents, actions, and their associations across multiple events. We adopt the Video-Language Entailment setup and propose StrictVLE that requires correct classification (rather than ranking) of the positive and negative caption. We evaluate several models and observe that even the best, LLaVA-OneVision (44.5%) and Gemini-1.5-Pro (49.3%), are far from human accuracy at 93.0%. Results show that action understanding lags behind agents, and negative captions created using entities appearing in the video perform worse than those obtained from pure text manipulation. We also present challenges with ClassicVLE and multiple-choice (MC) evaluation, strengthening our preference for StrictVLE. Finally, we validate that our benchmark requires visual inputs of multiple frames making it ideal to study video-language compositional reasoning.