Dynamic Memory Networks for Visual and Textual Question Answering
This work addresses the challenge of multimodal reasoning for AI systems, offering incremental improvements to existing memory-based architectures.
The authors tackled the problem of question answering across visual and textual modalities by improving the dynamic memory network (DMN) architecture, resulting in state-of-the-art performance on the Visual Question Answering and bAbI-10k datasets without requiring supporting fact supervision.
Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.