Task-driven Visual Saliency and Attention-based Visual Question Answering
This work improves VQA systems for applications requiring visual-textual understanding, but it is incremental as it builds on existing attention mechanisms.
The paper tackles the problem of visual question answering (VQA) by addressing limitations in current attention-based methods, which lose useful information by using single-value attention, and proposes a method with saliency pre-selection and element-wise multiplication attention, achieving strong empirical results on the COCO-VQA dataset.
Visual question answering (VQA) has witnessed great progress since May, 2015 as a classic problem unifying visual and textual data into a system. Many enlightening VQA works explore deep into the image and question encodings and fusing methods, of which attention is the most effective and infusive mechanism. Current attention based methods focus on adequate fusion of visual and textual features, but lack the attention to where people focus to ask questions about the image. Traditional attention based methods attach a single value to the feature at each spatial location, which losses many useful information. To remedy these problems, we propose a general method to perform saliency-like pre-selection on overlapped region features by the interrelation of bidirectional LSTM (BiLSTM), and use a novel element-wise multiplication based attention method to capture more competent correlation information between visual and textual features. We conduct experiments on the large-scale COCO-VQA dataset and analyze the effectiveness of our model demonstrated by strong empirical results.