Multi-modal Reference Learning for Fine-grained Text-to-Image Retrieval
This work addresses the challenge of ambiguous textual descriptions in fine-grained retrieval tasks, offering a novel approach for researchers and practitioners in computer vision and multimedia retrieval.
The paper tackles the problem of fine-grained text-to-image retrieval by addressing text ambiguity in training data, proposing a multi-modal reference learning framework that aggregates visual and textual details to improve representation learning and retrieval similarity, achieving a Rank1 accuracy of 56.2% on the RSTPReid dataset, surpassing the previous state-of-the-art by 5.6%.
Fine-grained text-to-image retrieval aims to retrieve a fine-grained target image with a given text query. Existing methods typically assume that each training image is accurately depicted by its textual descriptions. However, textual descriptions can be ambiguous and fail to depict discriminative visual details in images, leading to inaccurate representation learning. To alleviate the effects of text ambiguity, we propose a Multi-Modal Reference learning framework to learn robust representations. We first propose a multi-modal reference construction module to aggregate all visual and textual details of the same object into a comprehensive multi-modal reference. The multi-modal reference hence facilitates the subsequent representation learning and retrieval similarity computation. Specifically, a reference-guided representation learning module is proposed to use multi-modal references to learn more accurate visual and textual representations. Additionally, we introduce a reference-based refinement method that employs the object references to compute a reference-based similarity that refines the initial retrieval results. Extensive experiments are conducted on five fine-grained text-to-image retrieval datasets for different text-to-image retrieval tasks. The proposed method has achieved superior performance over state-of-the-art methods. For instance, on the text-to-person image retrieval dataset RSTPReid, our method achieves the Rank1 accuracy of 56.2\%, surpassing the recent CFine by 5.6\%.