Tailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillation
This work aims to improve the performance and convergence of federated learning for privacy-sensitive domains like healthcare, which is an incremental improvement for FL practitioners.
This paper tackles client heterogeneity in federated learning by proposing an optimization algorithm that integrates model delta regularization, personalized models, federated knowledge distillation, and mix-pooling. The experimental results show remarkable accuracy and rapid convergence with model delta regularization, improved FL performance with federated knowledge distillation in diverse data scenarios, and tangible benefits for clients from mix-pooling readout operations.
Federated learning (FL) has emerged as a transformative training paradigm, particularly invaluable in privacy-sensitive domains like healthcare. However, client heterogeneity in data, computing power, and tasks poses a significant challenge. To address such a challenge, we propose an FL optimization algorithm that integrates model delta regularization, personalized models, federated knowledge distillation, and mix-pooling. Model delta regularization optimizes model updates centrally on the server, efficiently updating clients with minimal communication costs. Personalized models and federated knowledge distillation strategies are employed to tackle task heterogeneity effectively. Additionally, mix-pooling is introduced to accommodate variations in the sensitivity of readout operations. Experimental results demonstrate the remarkable accuracy and rapid convergence achieved by model delta regularization. Additionally, the federated knowledge distillation algorithm notably improves FL performance, especially in scenarios with diverse data. Moreover, mix-pooling readout operations provide tangible benefits for clients, showing the effectiveness of our proposed methods.