AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent
This addresses the need for reduced user effort in hyperparameter tuning for practitioners using SGD, though it is incremental as it builds on existing SGD methods.
The paper tackles the problem of manually tuning learning rate schedules in stochastic gradient descent by introducing AutoSGD, which automatically adjusts the learning rate at each iteration, and shows strong empirical performance on various optimization and machine learning tasks.
The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks.