Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
This work addresses the restrictive answer formats in VQA by enabling more natural, multi-word responses with justifications, which is an incremental advancement for multimodal AI systems.
The authors tackled the limitation of existing Visual Question Answering models by proposing a generative approach that produces multi-word answers and rationales, introducing the ViQAR task, and demonstrating strong performance through quantitative evaluation and a human Turing Test.
Visual Question Answering is a multi-modal task that aims to measure high-level visual understanding. Contemporary VQA models are restrictive in the sense that answers are obtained via classification over a limited vocabulary (in the case of open-ended VQA), or via classification over a set of multiple-choice-type answers. In this work, we present a completely generative formulation where a multi-word answer is generated for a visual query. To take this a step forward, we introduce a new task: ViQAR (Visual Question Answering and Reasoning), wherein a model must generate the complete answer and a rationale that seeks to justify the generated answer. We propose an end-to-end architecture to solve this task and describe how to evaluate it. We show that our model generates strong answers and rationales through qualitative and quantitative evaluation, as well as through a human Turing Test.