Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks
This work addresses the need for more interpretable and interactive medical AI systems that can mimic doctors' region-focused analysis, though it represents an incremental improvement over existing medical MLLMs.
The paper tackles the problem that current medical multimodal large language models (MLLMs) struggle to identify specific anatomical regions in medical scans, which limits interpretability. By developing a region-aware bilingual MLLM called MedRegA and training it on a new dataset (MedRegInstruct), the model achieves state-of-the-art performance across 8 medical modalities for tasks like visual question answering and report generation.
Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a sentence. To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans. To achieve it, we first formulate Region-Centric tasks and construct a large-scale dataset, MedRegInstruct, to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a Region-Aware medical MLLM, MedRegA, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. Our project page is https://medrega.github.io.