Fast Asynchronous Parallel Stochastic Gradient Decent
This work addresses the need for faster and more efficient parallel optimization algorithms for large-scale machine learning applications, representing an incremental improvement over existing methods.
The paper tackles the problem of inefficient parallel stochastic gradient descent (SGD) methods for large-scale machine learning by proposing AsySVRG, an asynchronous parallel SGD method based on SVRG, which outperforms state-of-the-art methods like Hogwild! in convergence rate and computation cost.
Stochastic gradient descent~(SGD) and its variants have become more and more popular in machine learning due to their efficiency and effectiveness. To handle large-scale problems, researchers have recently proposed several parallel SGD methods for multicore systems. However, existing parallel SGD methods cannot achieve satisfactory performance in real applications. In this paper, we propose a fast asynchronous parallel SGD method, called AsySVRG, by designing an asynchronous strategy to parallelize the recently proposed SGD variant called stochastic variance reduced gradient~(SVRG). Both theoretical and empirical results show that AsySVRG can outperform existing state-of-the-art parallel SGD methods like Hogwild! in terms of convergence rate and computation cost.