HySTER: A Hybrid Spatio-Temporal Event Reasoner
This work addresses the need for more transparent and robust reasoning in VideoQA, which is incremental by integrating symbolic methods into a deep learning framework.
The paper tackles the problem of complex temporal and causal reasoning in Video Question Answering (VideoQA) by introducing HySTER, a hybrid model that combines deep learning with symbolic AI, achieving state-of-the-art accuracy on the CLEVRER dataset.
The task of Video Question Answering (VideoQA) consists in answering natural language questions about a video and serves as a proxy to evaluate the performance of a model in scene sequence understanding. Most methods designed for VideoQA up-to-date are end-to-end deep learning architectures which struggle at complex temporal and causal reasoning and provide limited transparency in reasoning steps. We present the HySTER: a Hybrid Spatio-Temporal Event Reasoner to reason over physical events in videos. Our model leverages the strength of deep learning methods to extract information from video frames with the reasoning capabilities and explainability of symbolic artificial intelligence in an answer set programming framework. We define a method based on general temporal, causal and physics rules which can be transferred across tasks. We apply our model to the CLEVRER dataset and demonstrate state-of-the-art results in question answering accuracy. This work sets the foundations for the incorporation of inductive logic programming in the field of VideoQA.