Stochastic Gradient Descent with Preconditioned Polyak Step-size
This work addresses the need for automated learning rate tuning in machine learning optimization, offering a method that reduces manual hyperparameter adjustment for researchers and engineers, though it is incremental as it builds on existing SPS.
The paper tackles the problem of tuning learning rates in Stochastic Gradient Descent by extending Stochastic Gradient Descent with Polyak Step-size (SPS) to incorporate preconditioning techniques like Hutchinson's method, Adam, and AdaGrad, resulting in improved performance on badly scaled or ill-conditioned datasets.
Stochastic Gradient Descent (SGD) is one of the many iterative optimization methods that are widely used in solving machine learning problems. These methods display valuable properties and attract researchers and industrial machine learning engineers with their simplicity. However, one of the weaknesses of this type of methods is the necessity to tune learning rate (step-size) for every loss function and dataset combination to solve an optimization problem and get an efficient performance in a given time budget. Stochastic Gradient Descent with Polyak Step-size (SPS) is a method that offers an update rule that alleviates the need of fine-tuning the learning rate of an optimizer. In this paper, we propose an extension of SPS that employs preconditioning techniques, such as Hutchinson's method, Adam, and AdaGrad, to improve its performance on badly scaled and/or ill-conditioned datasets.