Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning
This addresses the problem of high inference costs from training separate models for each modality in biomedical imaging, though it appears incremental as it builds on existing LoRA and contrastive learning techniques.
The paper tackles the problem of multimodal biomedical image incremental learning (MBIIL), where a unified model must be trained incrementally across different modalities, and proposes MSLoRA-CR, which fine-tunes modality-specific LoRA modules with contrastive regularization. The method achieves a 1.88% improvement in overall performance compared to unconstrained incremental learning methods while maintaining computational efficiency.
Multimodal Biomedical Image Incremental Learning (MBIIL) is essential for handling diverse tasks and modalities in the biomedical domain, as training separate models for each modality or task significantly increases inference costs. Existing incremental learning methods focus on task expansion within a single modality, whereas MBIIL seeks to train a unified model incrementally across modalities. The MBIIL faces two challenges: I) How to preserve previously learned knowledge during incremental updates? II) How to effectively leverage knowledge acquired from existing modalities to support new modalities? To address these challenges, we propose MSLoRA-CR, a method that fine-tunes Modality-Specific LoRA modules while incorporating Contrastive Regularization to enhance intra-modality knowledge sharing and promote inter-modality knowledge differentiation. Our approach builds upon a large vision-language model (LVLM), keeping the pretrained model frozen while incrementally adapting new LoRA modules for each modality or task. Experiments on the incremental learning of biomedical images demonstrate that MSLoRA-CR outperforms both the state-of-the-art (SOTA) approach of training separate models for each modality and the general incremental learning method (incrementally fine-tuning LoRA). Specifically, MSLoRA-CR achieves a 1.88% improvement in overall performance compared to unconstrained incremental learning methods while maintaining computational efficiency. Our code is publicly available at https://github.com/VentusAislant/MSLoRA_CR.