Visual Question Answering as Reading Comprehension
This approach addresses the challenge of multimodal fusion in VQA for AI systems, enabling the use of large-scale text and NLP techniques, though it is incremental as it builds on existing reading comprehension methods.
The paper tackles visual question answering (VQA) by converting it into a machine reading comprehension problem, unifying input information through natural language to handle both observation-based and knowledge-based questions, achieving comparable performance to state-of-the-art methods on three benchmarks.
Visual question answering (VQA) demands simultaneous comprehension of both the image visual content and natural language questions. In some cases, the reasoning needs the help of common sense or general knowledge which usually appear in the form of text. Current methods jointly embed both the visual information and the textual feature into the same space. However, how to model the complex interactions between the two different modalities is not an easy task. In contrast to struggling on multimodal feature fusion, in this paper, we propose to unify all the input information by natural language so as to convert VQA into a machine reading comprehension problem. With this transformation, our method not only can tackle VQA datasets that focus on observation based questions, but can also be naturally extended to handle knowledge-based VQA which requires to explore large-scale external knowledge base. It is a step towards being able to exploit large volumes of text and natural language processing techniques to address VQA problem. Two types of models are proposed to deal with open-ended VQA and multiple-choice VQA respectively. We evaluate our models on three VQA benchmarks. The comparable performance with the state-of-the-art demonstrates the effectiveness of the proposed method.