Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation
This work addresses the need for automated radiology report generation in medical imaging, but it is incremental as it builds on existing methods for multimodal alignment.
The authors tackled the problem of generating radiology reports from chest X-rays by introducing a visual language model that integrates an image encoder with a fine-tuned LLM, achieving notable accuracy in report generation.
We introduce a radiology-focused visual language model designed to generate radiology reports from chest X-rays. Building on previous findings that large language models (LLMs) can acquire multimodal capabilities when aligned with pretrained vision encoders, we demonstrate similar potential with chest X-ray images. This integration enhances the ability of model to understand and describe chest X-ray images. Our model combines an image encoder with a fine-tuned LLM based on the Vicuna-7B architecture, enabling it to generate different sections of a radiology report with notable accuracy. The training process involves a two-stage approach: (i) initial alignment of chest X-ray features with the LLM (ii) followed by fine-tuning for radiology report generation.