Visual Question Answering with Memory-Augmented Networks
This addresses the challenge of heavy-tailed answer distributions in visual question answering, which is an incremental improvement for the VQA domain.
The paper tackles the problem of answering visual questions when correct answers are rare in training data by using a memory-augmented neural network that selectively attends to training exemplars, achieving favorable performance compared to state-of-the-art methods on two large-scale benchmarks.
In this paper, we exploit a memory-augmented neural network to predict accurate answers to visual questions, even when those answers occur rarely in the training set. The memory network incorporates both internal and external memory blocks and selectively pays attention to each training exemplar. We show that memory-augmented neural networks are able to maintain a relatively long-term memory of scarce training exemplars, which is important for visual question answering due to the heavy-tailed distribution of answers in a general VQA setting. Experimental results on two large-scale benchmark datasets show the favorable performance of the proposed algorithm with a comparison to state of the art.