Stochastic Ratios Tracking Algorithm for Large Scale Machine Learning Problems
This addresses the need for automated step length tuning in SGD-based algorithms, which is crucial for efficiency in machine learning applications, though it appears incremental as it builds on existing SGD frameworks.
The paper tackles the problem of adaptive step length selection in stochastic gradient descent (SGD) for large-scale machine learning, proposing a novel algorithm that produces step lengths comparable to manually tuned ones in tests on logistic regression and deep neural networks.
Many machine learning applications and tasks rely on the stochastic gradient descent (SGD) algorithm and its variants. Effective step length selection is crucial for the success of these algorithms, which has motivated the development of algorithms such as ADAM or AdaGrad. In this paper, we propose a novel algorithm for adaptive step length selection in the classical SGD framework, which can be readily adapted to other stochastic algorithms. Our proposed algorithm is inspired by traditional nonlinear optimization techniques and is supported by analytical findings. We show that under reasonable conditions, the algorithm produces step lengths in line with well-established theoretical requirements, and generates iterates that converge to a stationary neighborhood of a solution in expectation. We test the proposed algorithm on logistic regressions and deep neural networks and demonstrate that the algorithm can generate step lengths comparable to the best step length obtained from manual tuning.