Semantic-aware Dynamic Retrospective-Prospective Reasoning for Event-level Video Question Answering
This addresses the need for explicit semantic connections in video-based question answering, particularly at the event level, for researchers in video understanding and AI.
The paper tackles the problem of complex reasoning across video events for Event-Level Video Question Answering (EVQA) by proposing a semantic-aware dynamic retrospective-prospective reasoning approach, which achieves superior performance on the TrafficQA benchmark compared to previous state-of-the-art models.
Event-Level Video Question Answering (EVQA) requires complex reasoning across video events to obtain the visual information needed to provide optimal answers. However, despite significant progress in model performance, few studies have focused on using the explicit semantic connections between the question and visual information especially at the event level. There is need for using such semantic connections to facilitate complex reasoning across video frames. Therefore, we propose a semantic-aware dynamic retrospective-prospective reasoning approach for video-based question answering. Specifically, we explicitly use the Semantic Role Labeling (SRL) structure of the question in the dynamic reasoning process where we decide to move to the next frame based on which part of the SRL structure (agent, verb, patient, etc.) of the question is being focused on. We conduct experiments on a benchmark EVQA dataset - TrafficQA. Results show that our proposed approach achieves superior performance compared to previous state-of-the-art models. Our code will be made publicly available for research use.