SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger
This addresses noise in datasets for vision-language models, offering a novel method for cross-modal alignment, though it is incremental on existing CLIP frameworks.
The paper tackles the problem of noise in image-text pairs for vision-language pre-training by proposing SoftCLIP, which relaxes strict one-to-one alignment and uses intra-modal self-similarity for softer cross-modal alignment, resulting in a top-1 accuracy improvement of 6.8%/7.2% over CLIP on ImageNet zero-shot classification with CC3M/CC12M datasets.
During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.