AlignVE: Visual Entailment Recognition Based on Alignment Relations
This addresses the problem of improving accuracy in visual entailment tasks for AI researchers, though it is incremental as it builds on existing methods by focusing on relation inference.
The paper tackled visual entailment recognition by proposing AlignVE, a new architecture that models the relation between premise images and hypothesis texts as an alignment matrix, achieving 72.45% accuracy on the SNLI-VE dataset and outperforming previous content-based models.
Visual entailment (VE) is to recognize whether the semantics of a hypothesis text can be inferred from the given premise image, which is one special task among recent emerged vision and language understanding tasks. Currently, most of the existing VE approaches are derived from the methods of visual question answering. They recognize visual entailment by quantifying the similarity between the hypothesis and premise in the content semantic features from multi modalities. Such approaches, however, ignore the VE's unique nature of relation inference between the premise and hypothesis. Therefore, in this paper, a new architecture called AlignVE is proposed to solve the visual entailment problem with a relation interaction method. It models the relation between the premise and hypothesis as an alignment matrix. Then it introduces a pooling operation to get feature vectors with a fixed size. Finally, it goes through the fully-connected layer and normalization layer to complete the classification. Experiments show that our alignment-based architecture reaches 72.45\% accuracy on SNLI-VE dataset, outperforming previous content-based models under the same settings.