A Novel Attention-based Aggregation Function to Combine Vision and Language
This work addresses a bottleneck in multimodal AI for tasks like image captioning and visual question answering, though it appears incremental as it builds on existing attention mechanisms.
The paper tackled the problem of combining vision and language representations by proposing a novel fully-attentive reduction method, which improved performance on image-text matching and visual question answering tasks using COCO and VQA 2.0 datasets.
The joint understanding of vision and language has been recently gaining a lot of attention in both the Computer Vision and Natural Language Processing communities, with the emergence of tasks such as image captioning, image-text matching, and visual question answering. As both images and text can be encoded as sets or sequences of elements -- like regions and words -- proper reduction functions are needed to transform a set of encoded elements into a single response, like a classification or similarity score. In this paper, we propose a novel fully-attentive reduction method for vision and language. Specifically, our approach computes a set of scores for each element of each modality employing a novel variant of cross-attention, and performs a learnable and cross-modal reduction, which can be used for both classification and ranking. We test our approach on image-text matching and visual question answering, building fair comparisons with other reduction choices, on both COCO and VQA 2.0 datasets. Experimentally, we demonstrate that our approach leads to a performance increase on both tasks. Further, we conduct ablation studies to validate the role of each component of the approach.