AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization
For researchers in multimodal representation learning, this work introduces a principled way to handle the one-to-many mapping between images and texts, improving the quality of learned representations.
AspectCLIP addresses the information asymmetry between images and texts in contrastive language-image pretraining, where captions often describe only one aspect of an image, causing global consistency regularizers to distort representations. By partitioning samples into aspect-coherent clusters and applying cyclic consistency within clusters while using prototype-level comparisons across clusters, AspectCLIP achieves a more structured representation space and outperforms traditional methods on downstream tasks.
Contrastive Language-Image Pretraining learns a shared representation space through large-scale contrastive learning. However, existing methods that enforce global consistency regularization overlook a key challenge: the inherent information asymmetry between images and text: captions typically describe only one specific aspect of an image, thus images with similar visual content can be paired with completely divergent textual content and semantic information. Consequently, global regularizers inadvertently impose constraints between visually similar images whose captions describe divergent aspects, introducing semantic distortion into the representation space. We propose AspectCLIP, a framework that reformulates consistency regularization to respect this one-to-many structure. AspectCLIP first partitions training samples into attribute clusters based on textual similarity to identify aspect-coherent groups, then applies full cyclic consistency within each cluster while restricting cross-cluster regularization to prototype-level comparisons. This aspect-guided regularization enforces strict geometric alignment only when images and texts describe a consistent facet, while allowing flexibility across divergent aspects. Extensive experiments on downstream tasks demonstrate that AspectCLIP consistently outperforms traditional methods and achieves a more structured representation space.