Asynchronous Distributed Semi-Stochastic Gradient Optimization
This addresses the problem of efficient large-scale distributed machine learning for practitioners, offering an incremental improvement over existing asynchronous SGD methods.
The paper tackles the trade-off between convergence speed and solution quality in distributed stochastic gradient descent by proposing an asynchronous distributed algorithm with variance reduction, achieving linear convergence to the optimal solution with a constant learning rate and outperforming state-of-the-art methods in wall clock time and solution quality on Google Cloud Platform.
With the recent proliferation of large-scale learning problems,there have been a lot of interest on distributed machine learning algorithms, particularly those that are based on stochastic gradient descent (SGD) and its variants. However, existing algorithms either suffer from slow convergence due to the inherent variance of stochastic gradients, or have a fast linear convergence rate but at the expense of poorer solution quality. In this paper, we combine their merits by proposing a fast distributed asynchronous SGD-based algorithm with variance reduction. A constant learning rate can be used, and it is also guaranteed to converge linearly to the optimal solution. Experiments on the Google Cloud Computing Platform demonstrate that the proposed algorithm outperforms state-of-the-art distributed asynchronous algorithms in terms of both wall clock time and solution quality.