Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances
This work addresses the problem of optimizing CLIP performance for vision tasks by analyzing backbone effects, offering incremental improvements through synergy and combination methods.
The study investigated performance differences among CLIP backbone architectures like ViTs and ResNets, finding that normalizing representations leads to significant variations and that informed backbone selection can improve performance by over 20%, with a proposed method for combining predictions boosting performance by up to 6.34%.
Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various neural architectures, spanning Transformer-based models like Vision Transformers (ViTs) to Convolutional Networks (ConvNets) like ResNets, are trained with CLIP and serve as universal backbones across diverse vision tasks. Despite utilizing the same data and training objectives, the effectiveness of representations learned by these architectures raises a critical question. Our investigation explores the differences in CLIP performance among these backbone architectures, revealing significant disparities in their classifications. Notably, normalizing these representations results in substantial performance variations. Our findings showcase a remarkable possible synergy between backbone predictions that could reach an improvement of over 20% through informed selection of the appropriate backbone. Moreover, we propose a simple, yet effective approach to combine predictions from multiple backbones, leading to a notable performance boost of up to 6.34\%. We will release the code for reproducing the results.