Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
This work addresses the gap in global model performance for federated learning applications under heterogeneous conditions, which is an incremental improvement over prior focus on client models.
The paper tackles the problem of global model performance degradation in heterogeneous federated learning by analyzing how custom client models trained with knowledge distillation affect the global model, and proposes a new method combining knowledge distillation and learning without forgetting to improve personalized models, achieving performance comparable to FedAvg in homogeneous settings with concrete results in realistic scenarios with dropping clients.
The heterogeneity of hardware and data is a well-known and studied problem in the community of Federated Learning (FL) as running under heterogeneous settings. Recently, custom-size client models trained with Knowledge Distillation (KD) has emerged as a viable strategy for tackling the heterogeneity challenge. However, previous efforts in this direction are aimed at client model tuning rather than their impact onto the knowledge aggregation of the global model. Despite performance of global models being the primary objective of FL systems, under heterogeneous settings client models have received more attention. Here, we provide more insights into how the chosen approach for training custom client models has an impact on the global model, which is essential for any FL application. We show the global model can fully leverage the strength of KD with heterogeneous data. Driven by empirical observations, we further propose a new approach that combines KD and Learning without Forgetting (LwoF) to produce improved personalised models. We bring heterogeneous FL on pair with the mighty FedAvg of homogeneous FL, in realistic deployment scenarios with dropping clients.