Distributed SGD Generalizes Well Under Asynchrony
This work addresses the scalability problem in distributed machine learning systems for practitioners dealing with big data, though it is incremental as it builds on existing stability frameworks.
The paper tackles the generalization performance of distributed asynchronous stochastic gradient descent (SGD) in systems with communication delays, proving that it generalizes well with sufficient data and suggesting a reduced learning rate strategy to improve stability and reduce error, which is confirmed through numerical experiments.
The performance of fully synchronized distributed systems has faced a bottleneck due to the big data trend, under which asynchronous distributed systems are becoming a major popularity due to their powerful scalability. In this paper, we study the generalization performance of stochastic gradient descent (SGD) on a distributed asynchronous system. The system consists of multiple worker machines that compute stochastic gradients which are further sent to and aggregated on a common parameter server to update the variables, and the communication in the system suffers from possible delays. Under the algorithm stability framework, we prove that distributed asynchronous SGD generalizes well given enough data samples in the training optimization. In particular, our results suggest to reduce the learning rate as we allow more asynchrony in the distributed system. Such adaptive learning rate strategy improves the stability of the distributed algorithm and reduces the corresponding generalization error. Then, we confirm our theoretical findings via numerical experiments.