Revisiting the "Video" in Video-Language Understanding
This work addresses the issue of benchmarking and model design in video-language understanding for researchers, revealing that current benchmarks may be incremental and not adequately test temporal reasoning.
The paper tackles the problem of assessing whether video-language tasks truly require temporal understanding by proposing the atemporal probe (ATP) model, which shows that strong or state-of-the-art performance on benchmarks like video question answering and text-to-video retrieval can be achieved without event temporality, even compared to large-scale models.
What makes a video task uniquely suited for videos, beyond what can be understood from a single image? Building on recent progress in self-supervised image-language models, we revisit this question in the context of video and language tasks. We propose the atemporal probe (ATP), a new model for video-language analysis which provides a stronger bound on the baseline accuracy of multimodal models constrained by image-level understanding. By applying this model to standard discriminative video and language tasks, such as video question answering and text-to-video retrieval, we characterize the limitations and potential of current video-language benchmarks. We find that understanding of event temporality is often not necessary to achieve strong or state-of-the-art performance, even compared with recent large-scale video-language models and in contexts intended to benchmark deeper video-level understanding. We also demonstrate how ATP can improve both video-language dataset and model design. We describe a technique for leveraging ATP to better disentangle dataset subsets with a higher concentration of temporally challenging data, improving benchmarking efficacy for causal and temporal understanding. Further, we show that effectively integrating ATP into full video-level temporal models can improve efficiency and state-of-the-art accuracy.