Exploiting LMM-based knowledge for image classification tasks
This is an incremental improvement for image classification tasks, leveraging multimodal models to enhance accuracy.
The paper tackled image classification by using MiniGPT-4 to generate semantic descriptions and combining image and text embeddings, resulting in improved performance validated on three datasets.
In this paper we address image classification tasks leveraging knowledge encoded in Large Multimodal Models (LMMs). More specifically, we use the MiniGPT-4 model to extract semantic descriptions for the images, in a multimodal prompting fashion. In the current literature, vision language models such as CLIP, among other approaches, are utilized as feature extractors, using only the image encoder, for solving image classification tasks. In this paper, we propose to additionally use the text encoder to obtain the text embeddings corresponding to the MiniGPT-4-generated semantic descriptions. Thus, we use both the image and text embeddings for solving the image classification task. The experimental evaluation on three datasets validates the improved classification performance achieved by exploiting LMM-based knowledge.