Communication-Efficient Stochastic Distributed Learning
This work addresses communication efficiency in distributed learning for networks, offering incremental improvements over existing methods.
The paper tackles the problem of high communication costs and large datasets in distributed learning by proposing a novel algorithm based on distributed ADMM, which allows multiple local training steps and stochastic gradients. It shows convergence to stationary or optimal points, with a variant achieving exact convergence through variance reduction, and demonstrates acceleration in convergence compared to state-of-the-art methods.
We address distributed learning problems, both nonconvex and convex, over undirected networks. In particular, we design a novel algorithm based on the distributed Alternating Direction Method of Multipliers (ADMM) to address the challenges of high communication costs, and large datasets. Our design tackles these challenges i) by enabling the agents to perform multiple local training steps between each round of communications; and ii) by allowing the agents to employ stochastic gradients while carrying out local computations. We show that the proposed algorithm converges to a neighborhood of a stationary point, for nonconvex problems, and of an optimal point, for convex problems. We also propose a variant of the algorithm to incorporate variance reduction thus achieving exact convergence. We show that the resulting algorithm indeed converges to a stationary (or optimal) point, and moreover that local training accelerates convergence. We thoroughly compare the proposed algorithms with the state of the art, both theoretically and through numerical results.