Simulating Clinical AI Assistance using Multimodal LLMs: A Case Study in Diabetic Retinopathy
This work addresses the challenge of optimizing AI output formats for clinician-AI collaboration in diabetic retinopathy screening, with potential applications in low-resource settings, though it is incremental as it builds on existing MLLM capabilities.
The study tackled the problem of limited clinical trust and utility in diabetic retinopathy screening by evaluating multimodal large language models (MLLMs) to simulate AI assistance across different output types, finding that MedGemma outperformed GPT-4o in baseline detection and that GPT-4o achieved strong results (AUROC up to 0.96) when guided by MedGemma's descriptive outputs.
Diabetic retinopathy (DR) is a leading cause of blindness worldwide, and AI systems can expand access to fundus photography screening. Current FDA-cleared systems primarily provide binary referral outputs, where this minimal output may limit clinical trust and utility. Yet, determining the most effective output format to enhance clinician-AI performance is an empirical challenge that is difficult to assess at scale. We evaluated multimodal large language models (MLLMs) for DR detection and their ability to simulate clinical AI assistance across different output types. Two models were tested on IDRiD and Messidor-2: GPT-4o, a general-purpose MLLM, and MedGemma, an open-source medical model. Experiments included: (1) baseline evaluation, (2) simulated AI assistance with synthetic predictions, and (3) actual AI-to-AI collaboration where GPT-4o incorporated MedGemma outputs. MedGemma outperformed GPT-4o at baseline, achieving higher sensitivity and AUROC, while GPT-4o showed near-perfect specificity but low sensitivity. Both models adjusted predictions based on simulated AI inputs, but GPT-4o's performance collapsed with incorrect ones, whereas MedGemma remained more stable. In actual collaboration, GPT-4o achieved strong results when guided by MedGemma's descriptive outputs, even without direct image access (AUROC up to 0.96). These findings suggest MLLMs may improve DR screening pipelines and serve as scalable simulators for studying clinical AI assistance across varying output configurations. Open, lightweight models such as MedGemma may be especially valuable in low-resource settings, while descriptive outputs could enhance explainability and clinician trust in clinical workflows.