A Bayesian Federated Learning Framework with Online Laplace Approximation
This work is significant for researchers and practitioners in federated learning, as it tackles critical issues of aggregation errors and local forgetting that hinder model performance in distributed learning environments.
This paper proposes a novel federated learning framework that addresses large aggregation errors and severe local forgetting, particularly in heterogeneous data settings. It achieves state-of-the-art results on several benchmarks by using online Laplace approximation to approximate posteriors on both client and server sides.
Federated learning (FL) allows multiple clients to collaboratively learn a globally shared model through cycles of model aggregation and local model training, without the need to share data. Most existing FL methods train local models separately on different clients, and then simply average their parameters to obtain a centralized model on the server side. However, these approaches generally suffer from large aggregation errors and severe local forgetting, which are particularly bad in heterogeneous data settings. To tackle these issues, in this paper, we propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. On the server side, a multivariate Gaussian product mechanism is employed to construct and maximize a global posterior, largely reducing the aggregation errors induced by large discrepancies between local models. On the client side, a prior loss that uses the global posterior probabilistic parameters delivered from the server is designed to guide the local training. Binding such learning constraints from other clients enables our method to mitigate local forgetting. Finally, we achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.