CoCo-IR: Contextual Composed Image Retrieval
This work addresses the limitation of single-turn interactions in image retrieval systems, providing a method for users to progressively refine visual searches, which is a significant improvement for real-world applications requiring complex, iterative search capabilities.
This paper introduces Contextual Composed Image Retrieval (CoCo-IR), a new task enabling iterative refinement of image search results. The proposed LMM-based model, which generates Transformable Image Embeddings (TIE), achieves 39.4 mAP@5 on the single-turn CIRCO benchmark and 44.1 R@1 on 4-turn dialogues in the new CoCo-IR benchmark, significantly outperforming existing methods (28.2 4-turn R@1).
Current instruction-based image retrieval systems are powerful but limited to single-turn interactions, failing to capture the iterative nature of complex, real-world visual searches. To overcome this limitation, we introduce Contextual Composed Image Retrieval (CoCo-IR), a novel task that enables users to progressively refine search results through interactions. We address this new task by proposing a new model based on a Large Multimodal Model (LMM) that functions as a context-aware reasoner for CoCo-IR. Our model interprets the entire interaction history to generate Transformable Image Embeddings (TIE) that evolve across turns. To fuel the model training without expensive human annotations, we develop a fully autonomous, scalable data engine that leverages LMMs to generate high-quality contextual retrieval data, and uses model-guided verification to mine challenging hard negatives. Extensive experiments demonstrate that our approach establishes new state-of-the-art performance: We achieve 39.4 mAP@5 on the challenging single-turn benchmark CIRCO; furthermore, on our new CoCo-IR benchmark, our model maintains robust performance with 44.1 R@1 on 4-turn dialogues, dramatically outperforming existing methods (28.2 4-turn R@1) that fail to handle multi-turn context. Project page: https://CoCo-IR.github.io.