Generating Rationales in Visual Question Answering
This addresses the challenge of evaluating reasoning in VQA for AI systems, but it is incremental as it builds on existing models and datasets.
The paper tackles the problem of assessing reasoning in Visual Question Answering (VQA) by proposing a new task of rationale generation, using data from VCR, and shows that joint training of ViLBERT with GPT-2 improves commonsense understanding through quantitative and qualitative metrics.
Despite recent advances in Visual QuestionAnswering (VQA), it remains a challenge todetermine how much success can be attributedto sound reasoning and comprehension ability.We seek to investigate this question by propos-ing a new task ofrationale generation. Es-sentially, we task a VQA model with generat-ing rationales for the answers it predicts. Weuse data from the Visual Commonsense Rea-soning (VCR) task, as it contains ground-truthrationales along with visual questions and an-swers. We first investigate commonsense un-derstanding in one of the leading VCR mod-els, ViLBERT, by generating rationales frompretrained weights using a state-of-the-art lan-guage model, GPT-2. Next, we seek to jointlytrain ViLBERT with GPT-2 in an end-to-endfashion with the dual task of predicting the an-swer in VQA and generating rationales. Weshow that this kind of training injects com-monsense understanding in the VQA modelthrough quantitative and qualitative evaluationmetrics