SA-VQA: Structured Alignment of Visual and Semantic Representations for Visual Question Answering
This work addresses a key bottleneck in multi-modal AI for tasks like VQA, offering an incremental improvement over existing entity-level alignment methods.
The paper tackles the problem of aligning visual and semantic representations in Visual Question Answering by proposing structured alignments using graph representations, which improves reasoning performance and interpretability, achieving state-of-the-art results on the GQA dataset and competitive performance on VQA-v2 without pretraining.
Visual Question Answering (VQA) attracts much attention from both industry and academia. As a multi-modality task, it is challenging since it requires not only visual and textual understanding, but also the ability to align cross-modality representations. Previous approaches extensively employ entity-level alignments, such as the correlations between the visual regions and their semantic labels, or the interactions across question words and object features. These attempts aim to improve the cross-modality representations, while ignoring their internal relations. Instead, we propose to apply structured alignments, which work with graph representation of visual and textual content, aiming to capture the deep connections between the visual and textual modalities. Nevertheless, it is nontrivial to represent and integrate graphs for structured alignments. In this work, we attempt to solve this issue by first converting different modality entities into sequential nodes and the adjacency graph, then incorporating them for structured alignments. As demonstrated in our experimental results, such a structured alignment improves reasoning performance. In addition, our model also exhibits better interpretability for each generated answer. The proposed model, without any pretraining, outperforms the state-of-the-art methods on GQA dataset, and beats the non-pretrained state-of-the-art methods on VQA-v2 dataset.