A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
This addresses the time-consuming and resource-intensive process of learning rate tuning for machine learning practitioners, offering an incremental improvement through automated adaptation.
The paper tackles the problem of manually tuning learning rates in stochastic optimization by providing a probabilistic motivation for existing methods and introducing a meta-algorithm that automatically adapts learning rates, demonstrating robust performance across a range of initial values on deep learning benchmarks.
Machine learning practitioners invest significant manual and computational resources in finding suitable learning rates for optimization algorithms. We provide a probabilistic motivation, in terms of Gaussian inference, for popular stochastic first-order methods. As an important special case, it recovers the Polyak step with a general metric. The inference allows us to relate the learning rate to a dimensionless quantity that can be automatically adapted during training by a control algorithm. The resulting meta-algorithm is shown to adapt learning rates in a robust manner across a large range of initial values when applied to deep learning benchmark problems.