ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos
This work addresses the challenge of temporal reasoning in videos for AI researchers, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of AI models' limited ability to perform temporal reasoning across video segments by introducing ReXTime, a benchmark suite that includes 921 validation and 2,143 test samples, revealing a 14.3% accuracy gap between frontier large language models and human performance.
We introduce ReXTime, a benchmark designed to rigorously test AI models' ability to perform temporal reasoning within video events. Specifically, ReXTime focuses on reasoning across time, i.e. human-like understanding when the question and its corresponding answer occur in different video segments. This form of reasoning, requiring advanced understanding of cause-and-effect relationships across video segments, poses significant challenges to even the frontier multimodal large language models. To facilitate this evaluation, we develop an automated pipeline for generating temporal reasoning question-answer pairs, significantly reducing the need for labor-intensive manual annotations. Our benchmark includes 921 carefully vetted validation samples and 2,143 test samples, each manually curated for accuracy and relevance. Evaluation results show that while frontier large language models outperform academic models, they still lag behind human performance by a significant 14.3% accuracy gap. Additionally, our pipeline creates a training dataset of 9,695 machine generated samples without manual effort, which empirical studies suggest can enhance the across-time reasoning via fine-tuning.