Pix2Key: Controllable Open-Vocabulary Retrieval with Semantic Decomposition and Self-Supervised Visual Dictionary Learning
This work provides a more intent-aware and diverse retrieval method for users performing composed image retrieval, which is an incremental improvement over existing methods.
This paper tackles composed image retrieval (CIR), where a reference image and a text edit are used to find images with the requested change. Pix2Key represents queries and candidates as open-vocabulary visual dictionaries, improving Recall@10 by up to 3.2 points on the DFMM-Compose benchmark, with an additional 2.3-point gain from a self-supervised pretraining component.
Composed Image Retrieval (CIR) uses a reference image plus a natural-language edit to retrieve images that apply the requested change while preserving other relevant visual content. Classic fusion pipelines typically rely on supervised triplets and can lose fine-grained cues, while recent zero-shot approaches often caption the reference image and merge the caption with the edit, which may miss implicit user intent and return repetitive results. We present Pix2Key, which represents both queries and candidates as open-vocabulary visual dictionaries, enabling intent-aware constraint matching and diversity-aware reranking in a unified embedding space. A self-supervised pretraining component, V-Dict-AE, further improves the dictionary representation using only images, strengthening fine-grained attribute understanding without CIR-specific supervision. On the DFMM-Compose benchmark, Pix2Key improves Recall@10 up to 3.2 points, and adding V-Dict-AE yields an additional 2.3-point gain while improving intent consistency and maintaining high list diversity.