ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity
This work addresses image retrieval for users needing to modify or specify traits in visual searches, representing an incremental improvement over existing methods.
The paper tackled the problem of searching for images using a query composed of an example image and complementary text, achieving state-of-the-art results on several retrieval benchmarks without relying on side information, multi-level features, heavy pre-training, or large architectures.
An intuitive way to search for images is to use queries composed of an example image and a complementary text. While the first provides rich and implicit context for the search, the latter explicitly calls for new traits, or specifies how some elements of the example image should be changed to retrieve the desired target image. Current approaches typically combine the features of each of the two elements of the query into a single representation, which can then be compared to the ones of the potential target images. Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval. Taking inspiration from them, we exploit the specific relation of each query element with the targeted image and derive light-weight attention mechanisms which enable to mediate between the two complementary modalities. We validate our approach on several retrieval benchmarks, querying with images and their associated free-form text modifiers. Our method obtains state-of-the-art results without resorting to side information, multi-level features, heavy pre-training nor large architectures as in previous works.