On Convergence of Federated Averaging Langevin Dynamics
This work addresses uncertainty quantification in federated learning for distributed clients, but it is incremental as it builds on existing Langevin dynamics and federated averaging methods.
The authors tackled the problem of uncertainty quantification and mean predictions in federated learning by proposing a federated averaging Langevin algorithm (FA-LD) for non-i.i.d. data, analyzing its convergence for strongly log-concave distributions and showing how factors like noise and learning rates affect it, with insights into optimizing local updates to reduce communication costs.
We propose a federated averaging Langevin algorithm (FA-LD) for uncertainty quantification and mean predictions with distributed clients. In particular, we generalize beyond normal posterior distributions and consider a general class of models. We develop theoretical guarantees for FA-LD for strongly log-concave distributions with non-i.i.d data and study how the injected noise and the stochastic-gradient noise, the heterogeneity of data, and the varying learning rates affect the convergence. Such an analysis sheds light on the optimal choice of local updates to minimize communication costs. Important to our approach is that the communication efficiency does not deteriorate with the injected noise in the Langevin algorithms. In addition, we examine in our FA-LD algorithm both independent and correlated noise used over different clients. We observe there is a trade-off between the pairs among communication, accuracy, and data privacy. As local devices may become inactive in federated networks, we also show convergence results based on different averaging schemes where only partial device updates are available. In such a case, we discover an additional bias that does not decay to zero.