Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study
This work addresses the challenge of adapting general vision-language models to medical imaging, which could reduce data requirements for healthcare applications.
This paper investigates whether pretrained vision-language models can transfer knowledge to medical image understanding, finding that well-designed medical prompts are crucial for eliciting domain knowledge and enabling zero-shot recognition with fewer samples. Experiments on thirteen medical datasets show that these prompts significantly improve zero-shot performance and fine-tuned models surpass supervised models by a large margin.
The large-scale pre-trained vision language models (VLM) have shown remarkable domain transfer capability on natural images. However, it remains unknown whether this capability can also apply to the medical image domain. This paper thoroughly studies the knowledge transferability of pre-trained VLMs to the medical domain, where we show that well-designed medical prompts are the key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting with expressive attributes that are shared between domains, the VLM can carry the knowledge across domains and improve its generalization. This mechanism empowers VLMs to recognize novel objects with fewer or without image samples. Furthermore, to avoid the laborious manual designing process, we develop three approaches for automatic generation of medical prompts, which can inject expert-level medical knowledge and image-specific information into the prompts for fine-grained grounding. We conduct extensive experiments on thirteen different medical datasets across various modalities, showing that our well-designed prompts greatly improve the zero-shot performance compared to the default prompts, and our fine-tuned models surpass the supervised models by a significant margin.