IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
This work improves bi-directional retrieval for vision-language understanding, with incremental advancements in handling semantic diversity.
The paper tackled the problem of cross-modal image-text retrieval by addressing the uniform alignment of diverse semantic complexities, proposing an iterative matching method with recurrent attention memory that achieved state-of-the-art performance on benchmark datasets like Flickr8K, Flickr30K, and MS COCO.
Enabling bi-directional retrieval of images and texts is important for understanding the correspondence between vision and language. Existing methods leverage the attention mechanism to explore such correspondence in a fine-grained manner. However, most of them consider all semantics equally and thus align them uniformly, regardless of their diverse complexities. In fact, semantics are diverse (i.e. involving different kinds of semantic concepts), and humans usually follow a latent structure to combine them into understandable languages. It may be difficult to optimally capture such sophisticated correspondences in existing methods. In this paper, to address such a deficiency, we propose an Iterative Matching with Recurrent Attention Memory (IMRAM) method, in which correspondences between images and texts are captured with multiple steps of alignments. Specifically, we introduce an iterative matching scheme to explore such fine-grained correspondence progressively. A memory distillation unit is used to refine alignment knowledge from early steps to later ones. Experiment results on three benchmark datasets, i.e. Flickr8K, Flickr30K, and MS COCO, show that our IMRAM achieves state-of-the-art performance, well demonstrating its effectiveness. Experiments on a practical business advertisement dataset, named \Ads{}, further validates the applicability of our method in practical scenarios.