Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcare
This work addresses data scarcity and privacy issues in healthcare AI by enabling hospitals to collaboratively fine-tune models without sharing sensitive patient data, though it is incremental as it builds on existing federated learning and fine-tuning techniques.
The paper tackles the challenge of fine-tuning vision foundation models for healthcare tasks under data privacy constraints by proposing a personalized federated learning method that uses orthogonal LoRA adapters to separate general and client-specific knowledge, achieving competitive performance against existing federated fine-tuning methods.
Foundation models open up new possibilities for the use of AI in healthcare. However, even when pre-trained on health data, they still need to be fine-tuned for specific downstream tasks. Furthermore, although foundation models reduce the amount of training data required to achieve good performance, obtaining sufficient data is still a challenge. This is due, in part, to restrictions on sharing and aggregating data from different sources to protect patients' privacy. One possible solution to this is to fine-tune foundation models via federated learning across multiple participating clients (i.e., hospitals, clinics, etc.). In this work, we propose a new personalized federated fine-tuning method that learns orthogonal LoRA adapters to disentangle general and client-specific knowledge, enabling each client to fully exploit both their own data and the data of others. Our preliminary results on real-world federated medical imaging tasks demonstrate that our approach is competitive against current federated fine-tuning methods.