Resolving learning rates adaptively by locating Stochastic Non-Negative Associated Gradient Projection Points using line searches
This work addresses the challenge of learning rate tuning for researchers and practitioners in machine learning, offering a method to automate this process and improve training efficiency, though it is incremental as it builds on existing line search and gradient-based optimization techniques.
The study tackled the problem of manually tuning learning rates in stochastic neural network training by proposing gradient-only line searches that locate Non-negative Associative Gradient Projection Points (NN-GPPs) to adaptively resolve learning rates, resulting in increased convergence performance and elimination of expensive hyperparameter tuning.
Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradient-only line searches to resolve the learning rate for neural network training algorithms. Stochastic sub-sampling during training decreases computational cost and allows the optimization algorithms to progress over local minima. However, it also results in discontinuous cost functions. Minimization line searches are not effective in this context, as they use a vanishing derivative (first order optimality condition), which often do not exist in a discontinuous cost function and therefore converge to discontinuities as opposed to minima from the data trends. Instead, we base candidate solutions along a search direction purely on gradient information, in particular by a directional derivative sign change from negative to positive (a Non-negative Associative Gradient Projection Point (NN- GPP)). Only considering a sign change from negative to positive always indicates a minimum, thus NN-GPPs contain second order information. Conversely, a vanishing gradient is purely a first order condition, which may indicate a minimum, maximum or saddle point. This insight allows the learning rate of an algorithm to be reliably resolved as the step size along a search direction, increasing convergence performance and eliminating an otherwise expensive hyperparameter.