Common Data Properties Limit Object-Attribute Binding in CLIP
This work addresses a critical limitation in CLIP models for vision-language tasks, revealing that data issues, not just model architecture, are key to solving object-attribute binding, which is incremental but important for improving multi-modal AI applications.
The study investigated how common data properties like low attribute density, incomplete captions, and saliency bias hinder CLIP's ability to bind objects and attributes, finding that only when these properties are addressed does CLIP achieve near-perfect binding, with no improvement from scaling batch size or adding hard negatives.
Contrastive vision-language models like CLIP are used for a large variety of applications, such as zero-shot classification or as vision encoder for multi-modal models. Despite their popularity, their representations show major limitations. For instance, CLIP models learn bag-of-words representations and, as a consequence, fail to distinguish whether an image is of ``a yellow submarine and a blue bus'' or ``a blue submarine and a yellow bus''. Previous attempts to fix this issue added hard negatives during training or modified the architecture, but failed to resolve the problem in its entirety. We suspect that the missing insights to solve the binding problem for CLIP are hidden in arguably the most important part of learning algorithms: the data. In this work, we fill this gap by rigorously identifying the influence of data properties on CLIP's ability to learn binding using a synthetic dataset. We find that common properties of natural data such as low attribute density, incomplete captions, and the saliency bias, a tendency of human captioners to describe the object that is ``most salient'' to them, have a detrimental effect on binding performance. In contrast to common belief, we find that neither scaling the batch size, i.e., implicitly adding more hard negatives, nor explicitly creating hard negatives enables CLIP to learn reliable binding. Only when the data expresses our identified data properties does CLIP learn almost perfect binding.