Towards Multimodal Active Learning: Efficient Learning with Limited Paired Data
This addresses the practical bottleneck of costly cross-modal alignments in multimodal pipelines like CLIP and SigLIP, offering an efficient solution for scenarios with limited paired data.
The paper tackles the problem of high annotation cost in multimodal learning by introducing the first framework for multimodal active learning with unaligned data, which reduces annotation requirements by up to 40% on the ColorSwap dataset without accuracy loss.
Active learning (AL) is a principled strategy to reduce annotation cost in data-hungry deep learning. However, existing AL algorithms focus almost exclusively on unimodal data, overlooking the substantial annotation burden in multimodal learning. We introduce the first framework for multimodal active learning with unaligned data, where the learner must actively acquire cross-modal alignments rather than labels on pre-aligned pairs. This setting captures the practical bottleneck in modern multimodal pipelines such as CLIP and SigLIP, where unimodal features are easy to obtain but high-quality alignment is costly. We develop a new algorithm that combines uncertainty and diversity principles in a modality-aware design, achieves linear-time acquisition, and applies seamlessly to both pool-based and streaming-based settings. Extensive experiments on benchmark datasets demonstrate that our approach consistently reduces multimodal annotation cost while preserving performance; for instance, on the ColorSwap dataset it cuts annotation requirements by up to $40\%$ without loss in accuracy.