Automatic and Simultaneous Adjustment of Learning Rate and Momentum for Stochastic Gradient Descent
This addresses the time-consuming and theoretically unjustified manual tuning of hyperparameters for machine learning practitioners, though it is incremental as it builds on existing SGD methods.
The paper tackles the problem of costly manual hyperparameter tuning in Stochastic Gradient Descent by proposing a method to automatically adjust learning rate and momentum using gradient statistics, achieving performance matching exhaustive search in training Convolutional Neural Networks.
Stochastic Gradient Descent (SGD) methods are prominent for training machine learning and deep learning models. The performance of these techniques depends on their hyperparameter tuning over time and varies for different models and problems. Manual adjustment of hyperparameters is very costly and time-consuming, and even if done correctly, it lacks theoretical justification which inevitably leads to "rule of thumb" settings. In this paper, we propose a generic approach that utilizes the statistics of an unbiased gradient estimator to automatically and simultaneously adjust two paramount hyperparameters: the learning rate and momentum. We deploy the proposed general technique for various SGD methods to train Convolutional Neural Networks (CNN's). The results match the performance of the best settings obtained through an exhaustive search and therefore, removes the need for a tedious manual tuning.