Image Search with Text Feedback by Additive Attention Compositional Learning
This addresses a practical problem for e-commerce applications by enabling more effective image search with user feedback.
The paper tackles the problem of image retrieval with text feedback, where a source image and text modifications are used to find target images, and proposes Additive Attention Compositional Learning (AACL), which achieves state-of-the-art results on three large-scale datasets.
Effective image retrieval with text feedback stands to impact a range of real-world applications, such as e-commerce. Given a source image and text feedback that describes the desired modifications to that image, the goal is to retrieve the target images that resemble the source yet satisfy the given modifications by composing a multi-modal (image-text) query. We propose a novel solution to this problem, Additive Attention Compositional Learning (AACL), that uses a multi-modal transformer-based architecture and effectively models the image-text contexts. Specifically, we propose a novel image-text composition module based on additive attention that can be seamlessly plugged into deep neural networks. We also introduce a new challenging benchmark derived from the Shopping100k dataset. AACL is evaluated on three large-scale datasets (FashionIQ, Fashion200k, and Shopping100k), each with strong baselines. Extensive experiments show that AACL achieves new state-of-the-art results on all three datasets.