Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
This addresses the problem of high training costs and data requirements for medical MLLMs, enabling more accessible development in medical imaging tasks, though it is incremental as it builds on existing MLLM frameworks.
The paper tackles the challenge of adapting multimodal large language models (MLLMs) for medical visual grounding, where models must locate areas in medical images based on text descriptions, by proposing a parameter-efficient fine-tuning method that achieves competitive results on a public benchmark and significantly outperforms GPT-4v.
Multimodal Large Language Models (MLLMs) inherit the superior text understanding capabilities of LLMs and extend these capabilities to multimodal scenarios. These models achieve excellent results in the general domain of multimodal tasks. However, in the medical domain, the substantial training costs and the requirement for extensive medical data pose challenges to the development of medical MLLMs. Furthermore, due to the free-text form of answers, tasks such as visual grounding that need to produce output in a prescribed form become difficult for MLLMs. So far, there have been no medical MLLMs works in medical visual grounding area. For the medical vision grounding task, which involves identifying locations in medical images based on short text descriptions, we propose Parameter-efficient Fine-tuning medical multimodal large language models for Medcial Visual Grounding (PFMVG). To validate the performance of the model, we evaluate it on a public benchmark dataset for medical visual grounding, where it achieves competitive results, and significantly outperforming GPT-4v. Our code will be open sourced after peer review.