Medical Vision-Language Pre-Training for Brain Abnormalities
This work addresses the need for specialized multimodal AI in clinical settings, focusing on brain abnormalities, but is incremental as it adapts existing vision-language methods to a medical domain.
The authors tackled the lack of domain-specific vision-language models for medical applications by automatically collecting a brain image-text dataset from public resources like PubMed and constructing a tailored model for brain abnormalities, achieving high performance in evaluations.
Vision-language models have become increasingly powerful for tasks that require an understanding of both visual and linguistic elements, bridging the gap between these modalities. In the context of multimodal clinical AI, there is a growing need for models that possess domain-specific knowledge, as existing models often lack the expertise required for medical applications. In this paper, we take brain abnormalities as an example to demonstrate how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset from case reports and published journals and subsequently constructing a high-performance vision-language model tailored to specific medical tasks. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain. We evaluated the resulting model with quantitative and qualitative intrinsic evaluations. The resulting dataset and our code can be found here https://github.com/masoud-monajati/MedVL_pretraining_pipeline