Invariant Grounding for Video Question Answering
This addresses unreliable reasoning in VideoQA models, which is a domain-specific problem for video understanding tasks, and represents a novel method for a known bottleneck.
The paper tackles the problem of spurious correlations in Video Question Answering (VideoQA) models by proposing a new learning framework called Invariant Grounding for VideoQA (IGV), which improves reasoning ability by grounding question-critical scenes with invariant causal relations, leading to superior accuracy, visual explainability, and generalization on three benchmark datasets.
Video Question Answering (VideoQA) is the task of answering questions about a video. At its core is understanding the alignments between visual scenes in video and linguistic semantics in question to yield the answer. In leading VideoQA models, the typical learning objective, empirical risk minimization (ERM), latches on superficial correlations between video-question pairs and answers as the alignments. However, ERM can be problematic, because it tends to over-exploit the spurious correlations between question-irrelevant scenes and answers, instead of inspecting the causal effect of question-critical scenes. As a result, the VideoQA models suffer from unreliable reasoning. In this work, we first take a causal look at VideoQA and argue that invariant grounding is the key to ruling out the spurious correlations. Towards this end, we propose a new learning framework, Invariant Grounding for VideoQA (IGV), to ground the question-critical scene, whose causal relations with answers are invariant across different interventions on the complement. With IGV, the VideoQA models are forced to shield the answering process from the negative influence of spurious correlations, which significantly improves the reasoning ability. Experiments on three benchmark datasets validate the superiority of IGV in terms of accuracy, visual explainability, and generalization ability over the leading baselines.