FOR: Finetuning for Object Level Open Vocabulary Image Retrieval
This addresses the challenge of accurately retrieving images with objects based on textual queries for applications in large-scale datasets, offering a significant performance boost.
The paper tackles the problem of open-vocabulary image retrieval by proposing FOR, a method that finetunes CLIP on target datasets using closed-set labels, resulting in improvements of up to 8 mAP@50 points over state-of-the-art across three datasets.
As working with large datasets becomes standard, the task of accurately retrieving images containing objects of interest by an open set textual query gains practical importance. The current leading approach utilizes a pre-trained CLIP model without any adaptation to the target domain, balancing accuracy and efficiency through additional post-processing. In this work, we propose FOR: Finetuning for Object-centric Open-vocabulary Image Retrieval, which allows finetuning on a target dataset using closed-set labels while keeping the visual-language association crucial for open vocabulary retrieval. FOR is based on two design elements: a specialized decoder variant of the CLIP head customized for the intended task, and its coupling within a multi-objective training framework. Together, these design choices result in a significant increase in accuracy, showcasing improvements of up to 8 mAP@50 points over SoTA across three datasets. Additionally, we demonstrate that FOR is also effective in a semi-supervised setting, achieving impressive results even when only a small portion of the dataset is labeled.