Explainable Learning Rate Regimes for Stochastic Optimization
This addresses the bottleneck of computational expense and time in hyperparameter tuning for practitioners using stochastic optimization methods, though it appears incremental as it builds on existing second-order algorithms.
The paper tackles the problem of manually tuning learning rate schedules in stochastic optimization by proposing an automatic regime that adjusts the learning rate based on the intrinsic variation of stochastic gradients, showing efficiency, robustness, and scalability across algorithms like SGD, SGDM, and SIGNSGD on machine learning tasks.
Modern machine learning is trained by stochastic gradient descent (SGD), whose performance critically depends on how the learning rate (LR) is adjusted and decreased over time. Yet existing LR regimes may be intricate, or need to tune one or more additional hyper-parameters manually whose bottlenecks include huge computational expenditure, time and power in practice. This work, in a natural and direct manner, clarifies how LR should be updated automatically only according to the intrinsic variation of stochastic gradients. An explainable LR regime by leveraging stochastic second-order algorithms is developed, behaving a similar pattern to heuristic algorithms but implemented simply without any parameter tuning requirement, where it is of an automatic procedure that LR should increase (decrease) as the norm of stochastic gradients decreases (increases). The resulting LR regime shows its efficiency, robustness, and scalability in different classical stochastic algorithms, containing SGD, SGDM, and SIGNSGD, on machine learning tasks.