QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
This work addresses the problem of efficient and private statistical inference in federated learning for decentralized data, offering incremental improvements over existing methods.
The paper tackles the challenges of privacy, communication overhead, and statistical heterogeneity in federated learning by proposing a novel Bayesian algorithm called Quantised Langevin Stochastic Dynamics (QLSD), which extends Stochastic Gradient Langevin Dynamics with gradient compression and variance reduction, achieving convergence guarantees and improved performance on benchmarks.
The objective of Federated Learning (FL) is to perform statistical inference for data which are decentralised and stored locally on networked clients. FL raises many constraints which include privacy and data ownership, communication overhead, statistical heterogeneity, and partial client participation. In this paper, we address these problems in the framework of the Bayesian paradigm. To this end, we propose a novel federated Markov Chain Monte Carlo algorithm, referred to as Quantised Langevin Stochastic Dynamics which may be seen as an extension to the FL setting of Stochastic Gradient Langevin Dynamics, which handles the communication bottleneck using gradient compression. To improve performance, we then introduce variance reduction techniques, which lead to two improved versions coined \texttt{QLSD}$^{\star}$ and \texttt{QLSD}$^{++}$. We give both non-asymptotic and asymptotic convergence guarantees for the proposed algorithms. We illustrate their performances using various Bayesian Federated Learning benchmarks.