Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
This work addresses medical VQA for clinical decision-making, presenting an incremental improvement as a simple baseline method.
The paper tackled the problem of medical visual question answering for wound-care by developing a retrieval-augmented generation system that uses a general-purpose large language model with multimodal exemplars, achieving a 3rd place ranking with an average score of 41.37% in the MEDIQA-WV 2025 shared task.
Medical Visual Question Answering (MedVQA) enables natural language queries over medical images to support clinical decision-making and patient care. The MEDIQA-WV 2025 shared task addressed wound-care VQA, requiring systems to generate free-text responses and structured wound attributes from images and patient queries. We present the MasonNLP system, which employs a general-domain, instruction-tuned large language model with a retrieval-augmented generation (RAG) framework that incorporates textual and visual examples from in-domain data. This approach grounds outputs in clinically relevant exemplars, improving reasoning, schema adherence, and response quality across dBLEU, ROUGE, BERTScore, and LLM-based metrics. Our best-performing system ranked 3rd among 19 teams and 51 submissions with an average score of 41.37%, demonstrating that lightweight RAG with general-purpose LLMs -- a minimal inference-time layer that adds a few relevant exemplars via simple indexing and fusion, with no extra training or complex re-ranking -- provides a simple and effective baseline for multimodal clinical NLP tasks.