Normalized and Geometry-Aware Self-Attention Network for Image Captioning
This work addresses the problem of improving image captioning accuracy for AI applications, though it is incremental as it builds on existing self-attention networks.
The paper tackled the limitations of self-attention in image captioning by introducing Normalized Self-Attention (NSA) for internal normalization and Geometry-aware Self-Attention (GSA) to model object geometry, achieving superior results on the MS-COCO dataset compared to state-of-the-art approaches.
Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we propose Normalized Self-Attention (NSA), a reparameterization of SA that brings the benefits of normalization inside SA. While normalization is previously only applied outside SA, we introduce a novel normalization method and demonstrate that it is both possible and beneficial to perform it on the hidden activations inside SA. Second, to compensate for the major limit of Transformer that it fails to model the geometry structure of the input objects, we propose a class of Geometry-aware Self-Attention (GSA) that extends SA to explicitly and efficiently consider the relative geometry relations between the objects in the image. To construct our image captioning model, we combine the two modules and apply it to the vanilla self-attention network. We extensively evaluate our proposals on MS-COCO image captioning dataset and superior results are achieved when comparing to state-of-the-art approaches. Further experiments on three challenging tasks, i.e. video captioning, machine translation, and visual question answering, show the generality of our methods.