Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher
This work addresses privacy-preserving collaborative learning for heterogeneous clients in federated settings, representing an incremental improvement over existing federated distillation methods.
The paper tackles the problem of misleading knowledge sharing in federated distillation due to heterogeneous client data and lack of a well-trained teacher, proposing a selective knowledge sharing mechanism that enhances generalization and outperforms baseline methods.
While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated distillation (FD), built upon knowledge distillation--an effective technique for transferring knowledge from a teacher model to student models--emerges as an alternative paradigm, which provides enhanced privacy guarantees and addresses model heterogeneity. Nevertheless, challenges arise due to variations in local data distributions and the absence of a well-trained teacher model, which leads to misleading and ambiguous knowledge sharing that significantly degrades model performance. To address these issues, this paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD. It includes client-side selectors and a server-side selector to accurately and precisely identify knowledge from local and ensemble predictions, respectively. Empirical studies, backed by theoretical insights, demonstrate that our approach enhances the generalization capabilities of the FD framework and consistently outperforms baseline methods.