Distributed optimization: designed for federated learning
This provides a theoretical framework for federated learning optimization that generalizes existing methods, though it appears incremental in nature.
The paper tackles the challenge of developing distributed optimization algorithms for federated learning that work with diverse communication topologies, proposing an augmented Lagrangian-based framework that recovers classical methods like gradient descent. Numerical experiments show strong performance in large-scale settings with significant statistical heterogeneity across clients.
Federated learning (FL), as a distributed collaborative machine learning (ML) framework under privacy-preserving constraints, has garnered increasing research attention in cross-organizational data collaboration scenarios. This paper proposes a class of distributed optimization algorithms based on the augmented Lagrangian technique, designed to accommodate diverse communication topologies in both centralized and decentralized FL settings. Furthermore, we develop multiple termination criteria and parameter update mechanisms to enhance computational efficiency, accompanied by rigorous theoretical guarantees of convergence. By generalizing the augmented Lagrangian relaxation through the incorporation of proximal relaxation and quadratic approximation, our framework systematically recovers a broad of classical unconstrained optimization methods, including proximal algorithm, classic gradient descent, and stochastic gradient descent, among others. Notably, the convergence properties of these methods can be naturally derived within the proposed theoretical framework. Numerical experiments demonstrate that the proposed algorithm exhibits strong performance in large-scale settings with significant statistical heterogeneity across clients.