MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation
This work addresses the need for more adaptable and interpretable AI models in medicine, which is inherently multimodal and multitask, though it appears incremental as it builds on existing vision-language model paradigms.
The authors tackled the problem of limited generalizability and explainability in medical AI by introducing MedViLaM, a multimodal large language model that achieved strong performance across multiple medical tasks, frequently outperforming other generalist models by a significant margin.
Medicine is inherently multimodal and multitask, with diverse data modalities spanning text, imaging. However, most models in medical field are unimodal single tasks and lack good generalizability and explainability. In this study, we introduce MedViLaM, a unified vision-language model towards a generalist model for medical data that can flexibly encode and interpret various forms of medical data, including clinical language and imaging, all using the same set of model weights. To facilitate the creation of such multi-task model, we have curated MultiMedBench, a comprehensive pretaining dataset and benchmark consisting of several distinct tasks, i.e., continuous question-answering, multi-label disease classification, disease localization, generation and summarization of radiology reports. MedViLaM demonstrates strong performance across all MultiMedBench tasks, frequently outpacing other generalist models by a significant margin. Additionally, we present instances of zero-shot generalization to new medical concepts and tasks, effective transfer learning across different tasks, and the emergence of zero-shot medical reasoning.