Multimodal Continuous Visual Attention Mechanisms
This work addresses a specific bottleneck in computer vision for tasks like visual question answering, offering an incremental improvement over existing continuous attention methods.
The paper tackles the limitation of existing continuous visual attention mechanisms, which use unimodal densities and struggle with complex or non-contiguous regions of interest, by introducing a multimodal continuous attention mechanism based on mixtures of Gaussians. The result is competitive accuracy on the VQA-v2 dataset and more human-like attention maps in VQA-HAT, with improved interpretability in complex scenes.
Visual attention mechanisms are a key component of neural network models for computer vision. By focusing on a discrete set of objects or image regions, these mechanisms identify the most relevant features and use them to build more powerful representations. Recently, continuous-domain alternatives to discrete attention models have been proposed, which exploit the continuity of images. These approaches model attention as simple unimodal densities (e.g. a Gaussian), making them less suitable to deal with images whose region of interest has a complex shape or is composed of multiple non-contiguous patches. In this paper, we introduce a new continuous attention mechanism that produces multimodal densities, in the form of mixtures of Gaussians. We use the EM algorithm to obtain a clustering of relevant regions in the image, and a description length penalty to select the number of components in the mixture. Our densities decompose as a linear combination of unimodal attention mechanisms, enabling closed-form Jacobians for the backpropagation step. Experiments on visual question answering in the VQA-v2 dataset show competitive accuracies and a selection of regions that mimics human attention more closely in VQA-HAT. We present several examples that suggest how multimodal attention maps are naturally more interpretable than their unimodal counterparts, showing the ability of our model to automatically segregate objects from ground in complex scenes.