Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities
This addresses the need for versatile retinal analysis models in medical imaging by handling incremental modality integration, though it is incremental as it builds on existing foundation model concepts.
The paper tackles the problem of integrating multiple fundus imaging modalities into a single foundation model in dynamic environments where data arrives incrementally, proposing RetCoP, a continual vision-language pre-training framework that achieves the best generalization and lowest forgetting rate compared to other methods.
Traditional fundus image analysis models focus on single-modal tasks, ignoring fundus modality complementarity, which limits their versatility. Recently, retinal foundation models have emerged, but most still remain modality-specific. Integrating multiple fundus imaging modalities into a single foundation model is valuable. However, in dynamic environments, data from different modalities often arrive incrementally, necessitating continual pre-training. To address this, we propose RetCoP, the first continual vision-language pre-training framework in the fundus domain, which incrementally integrates image and text features from different imaging modalities into a single unified foundation model. To mitigate catastrophic forgetting in continual pre-training, we introduce a rehearsal strategy utilizing representative image-text pairs and an off-diagonal information distillation approach. The former allows the model to revisit knowledge from previous stages, while the latter explicitly preserves the alignment between image and text representations. Experiments show that RetCoP outperforms all the compared methods, achieving the best generalization and lowest forgetting rate. The code can be found at https://github.com/Yuang-Yao/RetCoP.