Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
This provides a new perspective for federated learning, potentially improving efficiency and performance in distributed machine learning settings, though it is incremental as it builds on existing methods like FedAvg.
The paper tackled federated learning by reformulating it as a posterior inference problem, developing FedPA, an efficient algorithm that generalizes FedAvg and achieved state-of-the-art results with faster convergence and better optima on four benchmarks.
Federated learning is typically approached as an optimization problem, where the goal is to minimize a global loss function by distributing computation across client devices that possess local data and specify different parts of the global objective. We present an alternative perspective and formulate federated learning as a posterior inference problem, where the goal is to infer a global posterior distribution by having client devices each infer the posterior of their local data. While exact inference is often intractable, this perspective provides a principled way to search for global optima in federated settings. Further, starting with the analysis of federated quadratic objectives, we develop a computation- and communication-efficient approximate posterior inference algorithm -- federated posterior averaging (FedPA). Our algorithm uses MCMC for approximate inference of local posteriors on the clients and efficiently communicates their statistics to the server, where the latter uses them to refine a global estimate of the posterior mode. Finally, we show that FedPA generalizes federated averaging (FedAvg), can similarly benefit from adaptive optimizers, and yields state-of-the-art results on four realistic and challenging benchmarks, converging faster, to better optima.