Neural Self Talk: Image Understanding via Continuous Questioning and Answering
This addresses the challenge of automated image interpretation for applications like AI assistants or robotics, though it appears incremental as it combines existing VQG and VQA techniques.
The paper tackles the problem of continuous image understanding by developing a system that actively generates questions about images and answers them, using a Visual Question Generation module and a Visual Question Answering module trained simultaneously on a dataset of images, questions, and answers. The method was evaluated subjectively via Amazon Mechanical Turk, showing its effectiveness.
In this paper we consider the problem of continuously discovering image contents by actively asking image based questions and subsequently answering the questions being asked. The key components include a Visual Question Generation (VQG) module and a Visual Question Answering module, in which Recurrent Neural Networks (RNN) and Convolutional Neural Network (CNN) are used. Given a dataset that contains images, questions and their answers, both modules are trained at the same time, with the difference being VQG uses the images as input and the corresponding questions as output, while VQA uses images and questions as input and the corresponding answers as output. We evaluate the self talk process subjectively using Amazon Mechanical Turk, which show effectiveness of the proposed method.