Data Roaming and Quality Assessment for Composed Image Retrieval
This work addresses dataset and method limitations in CoIR, a task important for users needing precise multimodal queries, but it is incremental as it builds on existing CoIR frameworks.
The authors tackled the problem of limited and flawed datasets for Composed Image Retrieval (CoIR) by introducing the LaSCo dataset, which is ten times larger than existing ones and improves performance in pre-training, even in zero-shot, and proposed a new baseline method, CASE, that outperforms state-of-the-art methods on benchmarks like FashionIQ and CIRR.
The task of Composed Image Retrieval (CoIR) involves queries that combine image and text modalities, allowing users to express their intent more effectively. However, current CoIR datasets are orders of magnitude smaller compared to other vision and language (V&L) datasets. Additionally, some of these datasets have noticeable issues, such as queries containing redundant modalities. To address these shortcomings, we introduce the Large Scale Composed Image Retrieval (LaSCo) dataset, a new CoIR dataset which is ten times larger than existing ones. Pre-training on our LaSCo, shows a noteworthy improvement in performance, even in zero-shot. Furthermore, we propose a new approach for analyzing CoIR datasets and methods, which detects modality redundancy or necessity, in queries. We also introduce a new CoIR baseline, the Cross-Attention driven Shift Encoder (CASE). This baseline allows for early fusion of modalities using a cross-attention module and employs an additional auxiliary task during training. Our experiments demonstrate that this new baseline outperforms the current state-of-the-art methods on established benchmarks like FashionIQ and CIRR.