Grounded Image Text Matching with Mismatched Relation Reasoning
It addresses a specific challenge in visual-linguistic AI for researchers, but is incremental as it builds on existing transformer-based models and tasks.
This paper tackles the problem of evaluating relation understanding in visual-linguistic models by introducing the GITM-MR task, which requires models to determine if text describes an image and localize mismatched parts, and finds that pre-trained models lack data efficiency and length generalization ability.
This paper introduces Grounded Image Text Matching with Mismatched Relation (GITM-MR), a novel visual-linguistic joint task that evaluates the relation understanding capabilities of transformer-based pre-trained models. GITM-MR requires a model to first determine if an expression describes an image, then localize referred objects or ground the mismatched parts of the text. We provide a benchmark for evaluating pre-trained models on this task, with a focus on the challenging settings of limited data and out-of-distribution sentence lengths. Our evaluation demonstrates that pre-trained models lack data efficiency and length generalization ability. To address this, we propose the Relation-sensitive Correspondence Reasoning Network (RCRN), which incorporates relation-aware reasoning via bi-directional message propagation guided by language structure. RCRN can be interpreted as a modular program and delivers strong performance in both length generalization and data efficiency.