FedSplit: An algorithmic framework for fast federated optimization
This addresses a foundational issue in federated learning for distributed systems, offering a provably correct solution, though it appears incremental as it builds on operator splitting methods.
The paper tackles the problem of federated optimization where existing methods may not converge to correct stationary points, and introduces FedSplit, a class of algorithms that provably converge to optima with characterized rates and robustness to inexact computations.
Motivated by federated learning, we consider the hub-and-spoke model of distributed optimization in which a central authority coordinates the computation of a solution among many agents while limiting communication. We first study some past procedures for federated optimization, and show that their fixed points need not correspond to stationary points of the original optimization problem, even in simple convex settings with deterministic updates. In order to remedy these issues, we introduce FedSplit, a class of algorithms based on operator splitting procedures for solving distributed convex minimization with additive structure. We prove that these procedures have the correct fixed points, corresponding to optima of the original optimization problem, and we characterize their convergence rates under different settings. Our theory shows that these methods are provably robust to inexact computation of intermediate local quantities. We complement our theory with some simple experiments that demonstrate the benefits of our methods in practice.