Cooperative SGD with Dynamic Mixing Matrices
This work addresses inefficiencies in distributed machine learning for edge devices, though it appears incremental as it builds on existing SGD methods.
The paper tackles the problem of distributed SGD with fixed topologies and uniform aggregation, which is suboptimal, by proposing a unified framework for Local-Update SGD with dynamic topologies and non-uniform aggregation, resulting in improved or matching theoretical convergence guarantees.
One of the most common methods to train machine learning algorithms today is the stochastic gradient descent (SGD). In a distributed setting, SGD-based algorithms have been shown to converge theoretically under specific circumstances. A substantial number of works in the distributed SGD setting assume a fixed topology for the edge devices. These papers also assume that the contribution of nodes to the global model is uniform. However, experiments have shown that such assumptions are suboptimal and a non uniform aggregation strategy coupled with a dynamically shifting topology and client selection can significantly improve the performance of such models. This paper details a unified framework that covers several Local-Update SGD-based distributed algorithms with dynamic topologies and provides improved or matching theoretical guarantees on convergence compared to existing work.