Visual Robustness Benchmark for Visual Question Answering (VQA)
This addresses the problem of unreliable VQA systems in real-world applications, such as medical VQA, by providing a benchmark for assessing visual robustness, though it is incremental as it extends existing robustness work from linguistic to visual domains.
The authors tackled the lack of visual robustness evaluation in Visual Question Answering (VQA) systems by proposing the first large-scale benchmark with 213,000 augmented images, revealing insights into model performance and robustness under realistic corruptions like image blur.
Can Visual Question Answering (VQA) systems perform just as well when deployed in the real world? Or are they susceptible to realistic corruption effects e.g. image blur, which can be detrimental in sensitive applications, such as medical VQA? While linguistic or textual robustness has been thoroughly explored in the VQA literature, there has yet to be any significant work on the visual robustness of VQA models. We propose the first large-scale benchmark comprising 213,000 augmented images, challenging the visual robustness of multiple VQA models and assessing the strength of realistic visual corruptions. Additionally, we have designed several robustness evaluation metrics that can be aggregated into a unified metric and tailored to fit a variety of use cases. Our experiments reveal several insights into the relationships between model size, performance, and robustness with the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising the robustness.