WeaQA: Weak Supervision via Captions for Visual Question Answering
This work addresses the problem of data dependency and generalization for VQA models, which is crucial for researchers and practitioners aiming to reduce annotation costs and improve model robustness.
This paper explores training Visual Question Answering (VQA) models without human-annotated Image-Question-Answer (I-Q-A) triplets, instead using images and associated textual captions. They propose a method to train models with synthetically generated Q-A pairs and demonstrate its efficacy, particularly on the VQA-CP challenge which assesses performance under shifting linguistic priors.
Methodologies for training visual question answering (VQA) models assume the availability of datasets with human-annotated \textit{Image-Question-Answer} (I-Q-A) triplets. This has led to heavy reliance on datasets and a lack of generalization to new types of questions and scenes. Linguistic priors along with biases and errors due to annotator subjectivity have been shown to percolate into VQA models trained on such samples. We study whether models can be trained without any human-annotated Q-A pairs, but only with images and their associated textual descriptions or captions. We present a method to train models with synthetic Q-A pairs generated procedurally from captions. Additionally, we demonstrate the efficacy of spatial-pyramid image patches as a simple but effective alternative to dense and costly object bounding box annotations used in existing VQA models. Our experiments on three VQA benchmarks demonstrate the efficacy of this weakly-supervised approach, especially on the VQA-CP challenge, which tests performance under changing linguistic priors.