Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity
This work addresses a theoretical limitation in CLIP for researchers and practitioners, but it is incremental as it refines an existing framework.
The authors tackled the problem of suboptimal similarity computation in multi-modal contrastive pretraining like CLIP by proposing KME-CLIP, which leverages the linear structure of pointwise mutual information, resulting in overall performance improvements in retrieval and classification tasks.
In this study, we propose an enhancement to the similarity computation mechanism in multi-modal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel Hilbert space. We theoretically prove that our method can approximate PMI with arbitrary accuracy and empirically demonstrate that our approach overall outperforms the standard CLIP formulation across several retrieval and classification tasks.