Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuning
This addresses a challenging task in clinical practice for generating medical reports from images, but it is incremental as it builds on existing multimodal LLM approaches.
The paper tackles medical report generation from imaging data by introducing MRG-LLM, a multimodal large language model with a dynamic prompt customization mechanism, achieving state-of-the-art performance on IU X-ray and MIMIC-CXR datasets.
Medical report generation from imaging data remains a challenging task in clinical practice. While large language models (LLMs) show great promise in addressing this challenge, their effective integration with medical imaging data still deserves in-depth exploration. In this paper, we present MRG-LLM, a novel multimodal large language model (MLLM) that combines a frozen LLM with a learnable visual encoder and introduces a dynamic prompt customization mechanism. Our key innovation lies in generating instance-specific prompts tailored to individual medical images through conditional affine transformations derived from visual features. We propose two implementations: prompt-wise and promptbook-wise customization, enabling precise and targeted report generation. Extensive experiments on IU X-ray and MIMIC-CXR datasets demonstrate that MRG-LLM achieves state-of-the-art performance in medical report generation. Our code will be made publicly available.