Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insights
It addresses the problem of high resource costs for developing specialist medical VLMs, offering a scalable and cost-effective alternative for clinical AI development, though it is incremental in benchmarking existing methods.
This study compared generalist and specialist medical vision language models (VLMs) for clinical image diagnosis, finding that efficiently fine-tuned generalist VLMs can achieve comparable or superior performance in most tasks, especially on unseen or rare out-of-distribution medical modalities.
Vision Language Models (VLMs) have shown promise in automating image diagnosis and interpretation in clinical settings. However, developing specialist medical VLMs requires substantial computational resources and carefully curated datasets, and it remains unclear under which conditions generalist and specialist medical VLMs each perform best. This study highlights the complementary strengths of specialist medical and generalist VLMs. Specialists remain valuable in modality-aligned use cases, but we find that efficiently fine-tuned generalist VLMs can achieve comparable or even superior performance in most tasks, particularly when transferring to unseen or rare OOD medical modalities. These results suggest that generalist VLMs, rather than being constrained by their lack of specialist medical pretraining, may offer a scalable and cost-effective pathway for advancing clinical AI development.