Lightweight Joint Optimization of General-Purpose Vision-Language Models and Retrievers for RAG-Based Medical Diagnosis
This work addresses the challenge of enhancing medical diagnosis through retrieval-augmented generation (RAG) for clinicians, but it is incremental as it builds on existing general-purpose models with lightweight fine-tuning.
The paper tackles the problem of improving diagnostic accuracy in clinical image interpretation by jointly optimizing a multimodal retrieval model with a large vision-language model (LVLM) for medical diagnosis, achieving competitive results with medically-pretrained models on clinical classification and VQA tasks. It finds that joint optimization significantly improves challenging cases where different top-retrieved images lead to different predictions, though a large performance gap remains compared to an oracle.
Retrieving relevant visual and textual information from medical literature and hospital records can enhance diagnostic accuracy for clinical image interpretation. We develop a multimodal retrieval model jointly optimized with an LVLM for medical diagnosis, unlike standard RAG which doesn't backpropagate LVLM errors to the retriever. Using only general-purpose backbones with lightweight fine-tuning, our model achieves competitive results with medically-pretrained models on clinical classification and VQA tasks. In a novel analysis, we find that different top-retrieved images often yield different predictions for the same target, and that these cases are challenging for all models, even for non-retrieval models. Our joint retrieval optimization significantly improves these cases over standard RAG. However, oracle analysis reveals that while the correct diagnosis is frequently achievable using one of the top retrieved images, in practice there is a large performance gap from the oracle, and rerankers using frontier LVLMs do not close this gap -- leaving ample room for improvement by future methods. Code available at https://github.com/Nirmaz/JOMED.