Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workers
This addresses efficiency issues in large-scale distributed training for machine learning practitioners, though it is incremental as it builds on existing consensus-based methods.
The paper tackles the problem of stragglers slowing down consensus-based distributed machine learning by proposing a fully distributed algorithm that dynamically determines backup workers, achieving linear speedup in convergence as verified on MNIST and CIFAR-10 datasets.
With the increasing demand for large-scale training of machine learning models, consensus-based distributed optimization methods have recently been advocated as alternatives to the popular parameter server framework. In this paradigm, each worker maintains a local estimate of the optimal parameter vector, and iteratively updates it by waiting and averaging all estimates obtained from its neighbors, and then corrects it on the basis of its local dataset. However, the synchronization phase can be time consuming due to the need to wait for \textit{stragglers}, i.e., slower workers. An efficient way to mitigate this effect is to let each worker wait only for updates from the fastest neighbors before updating its local parameter. The remaining neighbors are called \textit{backup workers.} To minimize the globally training time over the network, we propose a fully distributed algorithm to dynamically determine the number of backup workers for each worker. We show that our algorithm achieves a linear speedup for convergence (i.e., convergence performance increases linearly with respect to the number of workers). We conduct extensive experiments on MNIST and CIFAR-10 to verify our theoretical results.