A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss
This addresses the challenge of non-IID data in federated learning for privacy-preserving distributed machine learning, representing an incremental improvement over existing personalized FL methods.
The paper tackled the problem of statistical heterogeneity in federated learning, which causes model drift and poor generalization, by proposing pFedKD-WCL, a novel algorithm that integrates knowledge distillation with bi-level optimization, and it outperformed state-of-the-art methods like FedAvg and FedProx in accuracy and convergence speed on MNIST and synthetic datasets.
Federated learning (FL) offers a privacy-preserving framework for distributed machine learning, enabling collaborative model training across diverse clients without centralizing sensitive data. However, statistical heterogeneity, characterized by non-independent and identically distributed (non-IID) client data, poses significant challenges, leading to model drift and poor generalization. This paper proposes a novel algorithm, pFedKD-WCL (Personalized Federated Knowledge Distillation with Weighted Combination Loss), which integrates knowledge distillation with bi-level optimization to address non-IID challenges. pFedKD-WCL leverages the current global model as a teacher to guide local models, optimizing both global convergence and local personalization efficiently. We evaluate pFedKD-WCL on the MNIST dataset and a synthetic dataset with non-IID partitioning, using multinomial logistic regression and multilayer perceptron models. Experimental results demonstrate that pFedKD-WCL outperforms state-of-the-art algorithms, including FedAvg, FedProx, Per-FedAvg, and pFedMe, in terms of accuracy and convergence speed.