A Lightweight Large Vision-language Model for Multimodal Medical Images
This work addresses the problem of resource-intensive medical VQA for clinical decision-making, though it is incremental as it builds on existing models like BiomedCLIP and LLaMA-3.
The paper tackled the challenge of developing efficient, high-performance medical visual question answering (VQA) models by introducing a lightweight multimodal model that integrates BiomedCLIP and LLaMA-3, achieving state-of-the-art performance with 73.4% accuracy on open-end questions on the OmniMedVQA dataset.
Medical Visual Question Answering (VQA) enhances clinical decision-making by enabling systems to interpret medical images and answer clinical queries. However, developing efficient, high-performance VQA models is challenging due to the complexity of medical imagery and diverse modalities. In this paper, we introduce a lightweight, multimodal VQA model integrating BiomedCLIP for image feature extraction and LLaMA-3 for text processing. Designed for medical VQA tasks, our model achieves state-of-the-art performance on the OmniMedVQA dataset. With approximately 8 billion parameters, it requires only two NVIDIA 40 GB A100 GPUs, demonstrating superior efficiency over larger models. Our results show 73.4% accuracy for open-end questions, surpassing existing models and validating its potential for real-world medical applications. Key contributions include a specialized multimodal VQA model, a resource-efficient architecture, and strong performance in answering open-ended clinical questions.