Hierarchical Conditional Relation Networks for Video Question Answering
This work addresses the problem of complex reasoning in video question answering for AI and computer vision researchers, presenting a novel method with incremental improvements in architecture design.
The paper tackled the challenge of video question answering by introducing a general-purpose neural unit called Conditional Relation Network (CRN) for representation and reasoning, achieving new state-of-the-art results on well-known datasets.
Video question answering (VideoQA) is challenging as it requires modeling capacity to distill dynamic visual artifacts and distant relations and to associate them with linguistic concepts. We introduce a general-purpose reusable neural unit called Conditional Relation Network (CRN) that serves as a building block to construct more sophisticated structures for representation and reasoning over video. CRN takes as input an array of tensorial objects and a conditioning feature, and computes an array of encoded output objects. Model building becomes a simple exercise of replication, rearrangement and stacking of these reusable units for diverse modalities and contextual information. This design thus supports high-order relational and multi-step reasoning. The resulting architecture for VideoQA is a CRN hierarchy whose branches represent sub-videos or clips, all sharing the same question as the contextual condition. Our evaluations on well-known datasets achieved new SoTA results, demonstrating the impact of building a general-purpose reasoning unit on complex domains such as VideoQA.