Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models
This addresses image retrieval for users needing keyword-based searches, but it is incremental as it adapts existing NLP techniques to a multi-modal context.
The paper tackles the problem of image retrieval by rethinking sparse lexical representations, using multi-modal large language models to convert image features into text for efficient retrieval algorithms, and shows superior precision and recall on datasets like MS-COCO, PASCAL VOC, and NUS-WIDE compared to conventional methods.
In this paper, we rethink sparse lexical representations for image retrieval. By utilizing multi-modal large language models (M-LLMs) that support visual prompting, we can extract image features and convert them into textual data, enabling us to utilize efficient sparse retrieval algorithms employed in natural language processing for image retrieval tasks. To assist the LLM in extracting image features, we apply data augmentation techniques for key expansion and analyze the impact with a metric for relevance between images and textual data. We empirically show the superior precision and recall performance of our image retrieval method compared to conventional vision-language model-based methods on the MS-COCO, PASCAL VOC, and NUS-WIDE datasets in a keyword-based image retrieval scenario, where keywords serve as search queries. We also demonstrate that the retrieval performance can be improved by iteratively incorporating keywords into search queries.