Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions
This work addresses the lack of standardized robustness analysis for VQA models, which is important for researchers and practitioners in computer vision and AI, though it is incremental as it builds on existing trends in adversarial evaluation.
The paper tackles the problem of evaluating robustness in Visual Question Answering (VQA) models by proposing a method that uses semantically related basic questions as noise, introducing a new robustness measure and datasets, and shows that in-context learning with these questions can improve model accuracy.
Deep neural networks have been critical in the task of Visual Question Answering (VQA), with research traditionally focused on improving model accuracy. Recently, however, there has been a trend towards evaluating the robustness of these models against adversarial attacks. This involves assessing the accuracy of VQA models under increasing levels of noise in the input, which can target either the image or the proposed query question, dubbed the main question. However, there is currently a lack of proper analysis of this aspect of VQA. This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models. It is hypothesized that as the similarity of a basic question to the main question decreases, the level of noise increases. To generate a reasonable noise level for a given main question, a pool of basic questions is ranked based on their similarity to the main question, and this ranking problem is cast as a LASSO optimization problem. Additionally, this work proposes a novel robustness measure, R_score, and two basic question datasets to standardize the analysis of VQA model robustness. The experimental results demonstrate that the proposed evaluation method effectively analyzes the robustness of VQA models. Moreover, the experiments show that in-context learning with a chain of basic questions can enhance model accuracy.