VITED: Video Temporal Evidence Distillation
This addresses the challenge of multi-step reasoning in video QA for applications requiring detailed temporal analysis, though it is incremental as it builds on existing datasets and models.
The paper tackles the problem of complex video question answering by introducing a framework for chain-of-evidence reasoning, which automatically constructs and uses temporal evidence spans to improve performance on long video QA benchmarks, outperforming state-of-the-art methods.
We investigate complex video question answering via chain-of-evidence reasoning -- identifying sequences of temporal spans from multiple relevant parts of the video, together with visual evidence within them. Existing models struggle with multi-step reasoning as they uniformly sample a fixed number of frames, which can miss critical evidence distributed nonuniformly throughout the video. Moreover, they lack the ability to temporally localize such evidence in the broader context of the full video, which is required for answering complex questions. We propose a framework to enhance existing VideoQA datasets with evidence reasoning chains, automatically constructed by searching for optimal intervals of interest in the video with supporting evidence, that maximizes the likelihood of answering a given question. We train our model (VITED) to generate these evidence chains directly, enabling it to both localize evidence windows as well as perform multi-step reasoning across them in long-form video content. We show the value of our evidence-distilled models on a suite of long video QA benchmarks where we outperform state-of-the-art approaches that lack evidence reasoning capabilities.