Knowledge Distillation for Federated Learning: a Practical Guide
This provides a practical guide for researchers and practitioners in federated learning to address data heterogeneity and efficiency challenges, though it is incremental as a review and classification effort.
The paper tackles the limitations of parameter-averaging federated learning algorithms, such as model homogeneity and high communication costs, by reviewing and classifying knowledge distillation-based approaches to mitigate these issues.
Federated Learning (FL) enables the training of Deep Learning models without centrally collecting possibly sensitive raw data. The most used algorithms for FL are parameter-averaging based schemes (e.g., Federated Averaging) that, however, have well known limits, i.e., model homogeneity, high communication cost, poor performance in presence of heterogeneous data distributions. Federated adaptations of regular Knowledge Distillation (KD) can solve or mitigate the weaknesses of parameter-averaging FL algorithms while possibly introducing other trade-offs. In this article, we originally present a focused review of the state-of-the-art KD-based algorithms specifically tailored for FL, by providing both a novel classification of the existing approaches and a detailed technical description of their pros, cons, and tradeoffs.