MultiRetNet: A Multimodal Vision Model and Deferral System for Staging Diabetic Retinopathy
This work addresses early detection of diabetic retinopathy, particularly in underserved populations, but appears incremental as it builds on existing multimodal and deferral approaches.
The authors tackled the problem of staging diabetic retinopathy by proposing MultiRetNet, a multimodal pipeline that combines retinal imaging, socioeconomic factors, and comorbidity profiles to improve staging accuracy, integrated with a deferral system for clinical review.
Diabetic retinopathy (DR) is a leading cause of preventable blindness, affecting over 100 million people worldwide. In the United States, individuals from lower-income communities face a higher risk of progressing to advanced stages before diagnosis, largely due to limited access to screening. Comorbid conditions further accelerate disease progression. We propose MultiRetNet, a novel pipeline combining retinal imaging, socioeconomic factors, and comorbidity profiles to improve DR staging accuracy, integrated with a clinical deferral system for a clinical human-in-the-loop implementation. We experiment with three multimodal fusion methods and identify fusion through a fully connected layer as the most versatile methodology. We synthesize adversarial, low-quality images and use contrastive learning to train the deferral system, guiding the model to identify out-of-distribution samples that warrant clinician review. By maintaining diagnostic accuracy on suboptimal images and integrating critical health data, our system can improve early detection, particularly in underserved populations where advanced DR is often first identified. This approach may reduce healthcare costs, increase early detection rates, and address disparities in access to care, promoting healthcare equity.