SimVQA: Exploring Simulated Environments for Visual Question Answering
This work addresses the problem of data diversity and privacy in VQA for AI researchers, offering an incremental improvement through synthetic data generation and feature augmentation.
The authors tackled the limited diversity in Visual Question Answering (VQA) datasets by using synthetic computer-generated data from 3D and physics simulations to control visual and language scenarios, showing that this approach generalizes to real-world VQA benchmarks and enhances domain invariance with a proposed Feature Swapping method.
Existing work on VQA explores data augmentation to achieve better generalization by perturbing the images in the dataset or modifying the existing questions and answers. While these methods exhibit good performance, the diversity of the questions and answers are constrained by the available image set. In this work we explore using synthetic computer-generated data to fully control the visual and language space, allowing us to provide more diverse scenarios. We quantify the effect of synthetic data in real-world VQA benchmarks and to which extent it produces results that generalize to real data. By exploiting 3D and physics simulation platforms, we provide a pipeline to generate synthetic data to expand and replace type-specific questions and answers without risking the exposure of sensitive or personal data that might be present in real images. We offer a comprehensive analysis while expanding existing hyper-realistic datasets to be used for VQA. We also propose Feature Swapping (F-SWAP) -- where we randomly switch object-level features during training to make a VQA model more domain invariant. We show that F-SWAP is effective for enhancing a currently existing VQA dataset of real images without compromising on the accuracy to answer existing questions in the dataset.