LMM-Regularized CLIP Embeddings for Image Classification
This work addresses image classification tasks for computer vision applications, presenting an incremental improvement by integrating LMM regularization into CLIP.
The paper tackles image classification by enhancing CLIP's image encoder with a Large Multimodal Model (LMM) based regularization method, which uses LMM-generated semantic descriptions to improve embedding discrimination, resulting in improved performance validated on three datasets.
In this paper we deal with image classification tasks using the powerful CLIP vision-language model. Our goal is to advance the classification performance using the CLIP's image encoder, by proposing a novel Large Multimodal Model (LMM) based regularization method. The proposed method uses an LMM to extract semantic descriptions for the images of the dataset. Then, it uses the CLIP's text encoder, frozen, in order to obtain the corresponding text embeddings and compute the mean semantic class descriptions. Subsequently, we adapt the CLIP's image encoder by adding a classification head, and we train it along with the image encoder output, apart from the main classification objective, with an additional auxiliary objective. The additional objective forces the embeddings at the image encoder's output to become similar to their corresponding LMM-generated mean semantic class descriptions. In this way, it produces embeddings with enhanced discrimination ability, leading to improved classification performance. The effectiveness of the proposed regularization method is validated through extensive experiments on three image classification datasets.