Med-LEGO: Editing and Adapting toward Generalist Medical Image Diagnosis
It addresses privacy and adaptability issues in computer-aided diagnosis for medical imaging, offering an incremental improvement over existing methods.
The paper tackles the challenge of creating generalist medical AI models without privacy-compromising pre-training by proposing Med-LEGO, a training-free framework that integrates specialist models like LEGO bricks, achieving outperformance in cross-domain and in-domain tasks with only 0.18% of full model parameters.
The adoption of visual foundation models has become a common practice in computer-aided diagnosis (CAD). While these foundation models provide a viable solution for creating generalist medical AI, privacy concerns make it difficult to pre-train or continuously update such models across multiple domains and datasets, leading many studies to focus on specialist models. To address this challenge, we propose Med-LEGO, a training-free framework that enables the seamless integration or updating of a generalist CAD model by combining multiple specialist models, similar to assembling LEGO bricks. Med-LEGO enhances LoRA (low-rank adaptation) by incorporating singular value decomposition (SVD) to efficiently capture the domain expertise of each specialist model with minimal additional parameters. By combining these adapted weights through simple operations, Med-LEGO allows for the easy integration or modification of specific diagnostic capabilities without the need for original data or retraining. Finally, the combined model can be further adapted to new diagnostic tasks, making it a versatile generalist model. Our extensive experiments demonstrate that Med-LEGO outperforms existing methods in both cross-domain and in-domain medical tasks while using only 0.18% of full model parameters. These merged models show better convergence and generalization to new tasks, providing an effective path toward generalist medical AI.