Automatic Synthesis of High-Quality Triplet Data for Composed Image Retrieval
This addresses the problem of costly manual data labeling for researchers and practitioners in vision-language tasks, enabling scalable and zero-shot CIR, though it is incremental in automating data synthesis.
The paper tackles the scalability and zero-shot capability limitations in Composed Image Retrieval (CIR) by proposing an automatic pipeline to generate high-quality synthetic triplets, resulting in a novel dataset (CIRHS) and a framework (CoAlign) that achieves outstanding zero-shot performance on benchmarks and outperforms state-of-the-art supervised methods.
As a challenging vision-language (VL) task, Composed Image Retrieval (CIR) aims to retrieve target images using multimodal (image+text) queries. Although many existing CIR methods have attained promising performance, their reliance on costly, manually labeled triplets hinders scalability and zero-shot capability. To address this issue, we propose a scalable pipeline for automatic triplet generation, along with a fully synthetic dataset named Composed Image Retrieval on High-quality Synthetic Triplets (CIRHS). Our pipeline leverages a large language model (LLM) to generate diverse prompts, controlling a text-to-image generative model to produce image pairs with identical elements in each pair, which are then filtered and reorganized to form the CIRHS dataset. In addition, we introduce Hybrid Contextual Alignment (CoAlign), a novel CIR framework, which can accomplish global alignment and local reasoning within a broader context, enabling the model to learn more robust and informative representations. By utilizing the synthetic CIRHS dataset, CoAlign achieves outstanding zero-shot performance on three commonly used benchmarks, demonstrating for the first time the feasibility of training CIR models on a fully synthetic dataset. Furthermore, under supervised training, our method outperforms all the state-of-the-art supervised CIR approaches, validating the effectiveness of our proposed retrieval framework. The code and the CIRHS dataset will be released soon.