Towards More Efficient Stochastic Decentralized Learning: Faster Convergence and Sparse Communication
This work addresses the problem of high communication costs and slow convergence in decentralized learning for researchers and practitioners, offering incremental improvements over existing methods.
The paper tackles the inefficiency of decentralized optimization by proposing DSBA, a stochastic algorithm that achieves geometric convergence with a linear dependence on the condition number and enables sparse communication, validated through experiments on convex minimization and AUC-maximization.
Recently, the decentralized optimization problem is attracting growing attention. Most existing methods are deterministic with high per-iteration cost and have a convergence rate quadratically depending on the problem condition number. Besides, the dense communication is necessary to ensure the convergence even if the dataset is sparse. In this paper, we generalize the decentralized optimization problem to a monotone operator root finding problem, and propose a stochastic algorithm named DSBA that (i) converges geometrically with a rate linearly depending on the problem condition number, and (ii) can be implemented using sparse communication only. Additionally, DSBA handles learning problems like AUC-maximization which cannot be tackled efficiently in the decentralized setting. Experiments on convex minimization and AUC-maximization validate the efficiency of our method.