Federated Learning with a Sampling Algorithm under Isoperimetry
This addresses the communication efficiency problem in federated learning for distributed device training, offering a novel Bayesian method that is more robust and informative, though it is incremental as it builds on existing Langevin sampling techniques.
The paper tackles the communication bottleneck in federated learning by proposing a Bayesian approach using a communication-efficient Langevin algorithm to sample from the posterior distribution, achieving robustness and providing more distributional knowledge compared to optimization methods, with analysis under the weaker log Sobolev inequality instead of strong log-concavity.
Federated learning uses a set of techniques to efficiently distribute the training of a machine learning algorithm across several devices, who own the training data. These techniques critically rely on reducing the communication cost -- the main bottleneck -- between the devices and a central server. Federated learning algorithms usually take an optimization approach: they are algorithms for minimizing the training loss subject to communication (and other) constraints. In this work, we instead take a Bayesian approach for the training task, and propose a communication-efficient variant of the Langevin algorithm to sample a posteriori. The latter approach is more robust and provides more knowledge of the \textit{a posteriori} distribution than its optimization counterpart. We analyze our algorithm without assuming that the target distribution is strongly log-concave. Instead, we assume the weaker log Sobolev inequality, which allows for nonconvexity.