Progressive Learning for Image Retrieval with Hybrid-Modality Queries
This addresses a challenging retrieval problem for domains like e-commerce where users need to search with combined image and text queries, representing an incremental advance with specific gains.
The paper tackles the problem of image retrieval with hybrid-modality queries (combining text and image), proposing a progressive learning approach that decomposes the task into three stages and includes a self-supervised adaptive weighting strategy. The result shows significant improvements, outperforming state-of-the-art methods by 24.9% and 9.5% in mean Recall@K on Fashion-IQ and Shoes datasets, respectively.
Image retrieval with hybrid-modality queries, also known as composing text and image for image retrieval (CTI-IR), is a retrieval task where the search intention is expressed in a more complex query format, involving both vision and text modalities. For example, a target product image is searched using a reference product image along with text about changing certain attributes of the reference image as the query. It is a more challenging image retrieval task that requires both semantic space learning and cross-modal fusion. Previous approaches that attempt to deal with both aspects achieve unsatisfactory performance. In this paper, we decompose the CTI-IR task into a three-stage learning problem to progressively learn the complex knowledge for image retrieval with hybrid-modality queries. We first leverage the semantic embedding space for open-domain image-text retrieval, and then transfer the learned knowledge to the fashion-domain with fashion-related pre-training tasks. Finally, we enhance the pre-trained model from single-query to hybrid-modality query for the CTI-IR task. Furthermore, as the contribution of individual modality in the hybrid-modality query varies for different retrieval scenarios, we propose a self-supervised adaptive weighting strategy to dynamically determine the importance of image and text in the hybrid-modality query for better retrieval. Extensive experiments show that our proposed model significantly outperforms state-of-the-art methods in the mean of Recall@K by 24.9% and 9.5% on the Fashion-IQ and Shoes benchmark datasets respectively.