Interpretable Visual Question Answering Referring to Outside Knowledge
This work addresses the need for more rational and diverse explanations in VQA systems, which is incremental as it builds on existing methods by integrating external knowledge.
The paper tackles the problem of visual question answering by developing a multimodal interpretable model that incorporates outside knowledge and multiple image captions to improve answer accuracy and explanation rationality, achieving state-of-the-art performance in both metrics.
We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model's decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.