Knowledge Rumination for Client Utility Evaluation in Heterogeneous Federated Learning
This work addresses practical issues in federated learning for clients with heterogeneous data, offering incremental improvements to enhance model stability and efficiency.
The paper tackles the challenges of Non-IID data and stale models in asynchronous federated learning by proposing FedHist, a framework that uses knowledge rumination and an intelligent ℓ₂-norm amplification scheme, resulting in improved convergence performance and test accuracy over state-of-the-art methods.
Federated Learning (FL) allows several clients to cooperatively train machine learning models without disclosing the raw data. In practical applications, asynchronous FL (AFL) can address the straggler effect compared to synchronous FL. However, Non-IID data and stale models pose significant challenges to AFL, as they can diminish the practicality of the global model and even lead to training failures. In this work, we propose a novel AFL framework called Federated Historical Learning (FedHist), which effectively addresses the challenges posed by both Non-IID data and gradient staleness based on the concept of knowledge rumination. FedHist enhances the stability of local gradients by performing weighted fusion with historical global gradients cached on the server. Relying on hindsight, it assigns aggregation weights to each participant in a multi-dimensional manner during each communication round. To further enhance the efficiency and stability of the training process, we introduce an intelligent $\ell_2$-norm amplification scheme, which dynamically regulates the learning progress based on the $\ell_2$-norms of the submitted gradients. Extensive experiments indicate FedHist outperforms state-of-the-art methods in terms of convergence performance and test accuracy.