MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model
This work addresses scaling and contextual limitations in multimodal AI for researchers and practitioners, offering an incremental improvement in training efficiency and representation coherence.
The paper tackles the computational inefficiency and lack of contextual modeling in single-turn contrastive learning for multimodal embedding models by introducing Multi-Turn Contrastive Learning (MuCo), which processes multiple query-target pairs per image in one forward pass, achieving state-of-the-art retrieval performance on benchmarks like MMEB and M-BEIR with a newly curated 5M dataset.
Universal Multimodal embedding models built on Multimodal Large Language Models (MLLMs) have traditionally employed contrastive learning, which aligns representations of query-target pairs across different modalities. Yet, despite its empirical success, they are primarily built on a "single-turn" formulation where each query-target pair is treated as an independent data point. This paradigm leads to computational inefficiency when scaling, as it requires a separate forward pass for each pair and overlooks potential contextual relationships between multiple queries that can relate to the same context. In this work, we introduce Multi-Turn Contrastive Learning (MuCo), a dialogue-inspired framework that revisits this process. MuCo leverages the conversational nature of MLLMs to process multiple, related query-target pairs associated with a single image within a single forward pass. This allows us to extract a set of multiple query and target embeddings simultaneously, conditioned on a shared context representation, amplifying the effective batch size and overall training efficiency. Experiments exhibit MuCo with a newly curated 5M multimodal multi-turn dataset (M3T), which yields state-of-the-art retrieval performance on MMEB and M-BEIR benchmarks, while markedly enhancing both training efficiency and representation coherence across modalities. Code and M3T are available at https://github.com/naver-ai/muco