Accelerated Almost-Sure Convergence Rates for Nonconvex Stochastic Gradient Descent using Stochastic Learning Rates
This work addresses efficiency in large-scale nonconvex optimization, which is incremental as it modifies the learning rate scheme rather than introducing a new paradigm.
The paper tackles the problem of improving convergence rates for Stochastic Gradient Descent (SGD) in nonconvex optimization by introducing a stochastic learning rate scheme, resulting in accelerated almost-sure convergence rates compared to deterministic learning rates, with theoretical results empirically verified.
Large-scale optimization problems require algorithms both effective and efficient. One such popular and proven algorithm is Stochastic Gradient Descent which uses first-order gradient information to solve these problems. This paper studies almost-sure convergence rates of the Stochastic Gradient Descent method when instead of deterministic, its learning rate becomes stochastic. In particular, its learning rate is equipped with a multiplicative stochasticity, producing a stochastic learning rate scheme. Theoretical results show accelerated almost-sure convergence rates of Stochastic Gradient Descent in a nonconvex setting when using an appropriate stochastic learning rate, compared to a deterministic-learning-rate scheme. The theoretical results are verified empirically.