Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogenous Federated Learning
This addresses efficiency and accuracy issues in federated learning for edge computing, though it is incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem of weak client participation in heterogeneous federated learning by proposing an on-device knowledge distillation method that leverages unlabeled data and system resources, achieving higher accuracy compared to state-of-the-art methods.
Online Knowledge Distillation (KD) is recently highlighted to train large models in Federated Learning (FL) environments. Many existing studies adopt the logit ensemble method to perform KD on the server side. However, they often assume that unlabeled data collected at the edge is centralized on the server. Moreover, the logit ensemble method personalizes local models, which can degrade the quality of soft targets, especially when data is highly non-IID. To address these critical limitations,we propose a novel on-device KD-based heterogeneous FL method. Our approach leverages a small auxiliary model to learn from labeled local data. Subsequently, a subset of clients with strong system resources transfers knowledge to a large model through on-device KD using their unlabeled data. Our extensive experiments demonstrate that our on-device KD-based heterogeneous FL method effectively utilizes the system resources of all edge devices as well as the unlabeled data, resulting in higher accuracy compared to SOTA KD-based FL methods.