Variational Adapter for Cross-modal Similarity Representation
This work is significant for researchers and practitioners working with vision-language models, particularly in image-text retrieval and generalization tasks, by addressing the limitations of binary annotations and improving model robustness.
The paper tackles the problem of limited fine-grained cross-modal matching annotations in vision-language models, which leads to false negative samples and impaired generalization. They propose the Variational Adapter for Cross-modal Similarity Representation (VACSR) to reformulate image-text matching as a variational inference problem, constructing a latent space for cross-modal similarity and using regularization to mitigate overfitting to binary annotations.
The core of vision-language models lies in measuring cross-modal similarity within a unified representation space. However, most image-text matching or multi-class image classification datasets lack fine-grained cross-modal matching annotations, forcing the continuous similarity space into binary classification boundaries. This compression induces false negative samples and significantly impairs the generalization performance of cross-modal tasks. While prior research has attempted to mitigate this by modeling intra-modal ambiguity, it often overlooks inherent annotation flaws, leading to suboptimal uncertainty allocation. To address these challenges, we propose a Variational Adapter for Cross-modal Similarity Representation (VACSR). This approach reformulates image-text matching with fine-grained semantic scarcity as a variational inference problem. It constructs a latent space for cross-modal similarity and uses regularization techniques to mitigate overfitting to binary annotations. Experiments on image-text retrieval, domain generalization, and base-to-novel generalization demonstrate the proposed method's effectiveness and robust generalization ability.