EKTVQA: Generalized use of External Knowledge to empower Scene Text in Text-VQA
It addresses training data bias and improves answer accuracy for scene text in visual question answering, but is incremental as it builds on existing multimodal transformers.
The paper tackles the zero-shot problem in Text-VQA by using external knowledge to augment scene text understanding, achieving results comparable to state-of-the-art on three datasets.
The open-ended question answering task of Text-VQA often requires reading and reasoning about rarely seen or completely unseen scene-text content of an image. We address this zero-shot nature of the problem by proposing the generalized use of external knowledge to augment our understanding of the scene text. We design a framework to extract, validate, and reason with knowledge using a standard multimodal transformer for vision language understanding tasks. Through empirical evidence and qualitative results, we demonstrate how external knowledge can highlight instance-only cues and thus help deal with training data bias, improve answer entity type correctness, and detect multiword named entities. We generate results comparable to the state-of-the-art on three publicly available datasets, under the constraints of similar upstream OCR systems and training data.