A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
This work addresses the challenge of visual question answering for AI systems, but it appears incremental as it combines existing NLP and CV advances without claiming major breakthroughs.
The authors tackled the problem of automatically answering complex questions about real-world images by proposing a multi-world approach that integrates uncertain visual predictions with discrete reasoning in a Bayesian framework, achieving a first benchmark for this task.
We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision. We combine discrete reasoning with uncertain predictions by a multi-world approach that represents uncertainty about the perceived world in a bayesian framework. Our approach can handle human questions of high complexity about realistic scenes and replies with range of answer like counts, object classes, instances and lists of them. The system is directly trained from question-answer pairs. We establish a first benchmark for this task that can be seen as a modern attempt at a visual turing test.