VQA-GEN: A Visual Question Answering Benchmark for Domain Generalization
This addresses the problem of limited real-world applicability for VQA models by providing a domain generalization benchmark, though it is incremental as it builds on existing domain generalization datasets.
The paper tackles the lack of comprehensive benchmark datasets for visual question answering (VQA) by introducing VQA-GEN, a multi-modal benchmark for domain generalization that exposes vulnerabilities of existing methods to joint visual and textual distribution shifts, with models trained on it showing improved cross-domain and in-domain performance.
Visual question answering (VQA) models are designed to demonstrate visual-textual reasoning capabilities. However, their real-world applicability is hindered by a lack of comprehensive benchmark datasets. Existing domain generalization datasets for VQA exhibit a unilateral focus on textual shifts while VQA being a multi-modal task contains shifts across both visual and textual domains. We propose VQA-GEN, the first ever multi-modal benchmark dataset for distribution shift generated through a shift induced pipeline. Experiments demonstrate VQA-GEN dataset exposes the vulnerability of existing methods to joint multi-modal distribution shifts. validating that comprehensive multi-modal shifts are critical for robust VQA generalization. Models trained on VQA-GEN exhibit improved cross-domain and in-domain performance, confirming the value of VQA-GEN. Further, we analyze the importance of each shift technique of our pipeline contributing to the generalization of the model.