Few-shot medical image classification with simple shape and texture text descriptors using vision-language models
This work addresses medical image analysis for healthcare applications, but it is incremental as it adapts existing models to a new domain.
The study tackled few-shot medical image classification by using GPT-4 to generate shape and texture text descriptors and applying vision-language models, achieving viable results for chest X-rays and breast ultrasound images, though requiring descriptor exclusion for accuracy.
In this work, we investigate the usefulness of vision-language models (VLMs) and large language models for binary few-shot classification of medical images. We utilize the GPT-4 model to generate text descriptors that encapsulate the shape and texture characteristics of objects in medical images. Subsequently, these GPT-4 generated descriptors, alongside VLMs pre-trained on natural images, are employed to classify chest X-rays and breast ultrasound images. Our results indicate that few-shot classification of medical images using VLMs and GPT-4 generated descriptors is a viable approach. However, accurate classification requires to exclude certain descriptors from the calculations of the classification scores. Moreover, we assess the ability of VLMs to evaluate shape features in breast mass ultrasound images. We further investigate the degree of variability among the sets of text descriptors produced by GPT-4. Our work provides several important insights about the application of VLMs for medical image analysis.