Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection
This work addresses the challenge of integrating external world knowledge into bi-modal VQA systems, offering insights for improving model understanding in entity-centric and common sense reasoning tasks, though it is incremental as it builds on existing knowledge injection methods.
The paper tackled the problem of Knowledge-Based Visual Question Answering (KBVQA) by empirically studying how entity-enhanced knowledge injection improves performance on existing VQA systems, showing substantial gains without additional costly pre-training on datasets like KVQA and OKVQA.
Knowledge-Based Visual Question Answering (KBVQA) is a bi-modal task requiring external world knowledge in order to correctly answer a text question and associated image. Recent single modality text work has shown knowledge injection into pre-trained language models, specifically entity enhanced knowledge graph embeddings, can improve performance on downstream entity-centric tasks. In this work, we empirically study how and whether such methods, applied in a bi-modal setting, can improve an existing VQA system's performance on the KBVQA task. We experiment with two large publicly available VQA datasets, (1) KVQA which contains mostly rare Wikipedia entities and (2) OKVQA which is less entity-centric and more aligned with common sense reasoning. Both lack explicit entity spans and we study the effect of different weakly supervised and manual methods for obtaining them. Additionally we analyze how recently proposed bi-modal and single modal attention explanations are affected by the incorporation of such entity enhanced representations. Our results show substantial improved performance on the KBVQA task without the need for additional costly pre-training and we provide insights for when entity knowledge injection helps improve a model's understanding. We provide code and enhanced datasets for reproducibility.