Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent
This addresses the challenge of tuning learning rates for practitioners in optimization, but it is incremental as it builds on existing gradient descent methods.
The paper tackles the problem of adaptively choosing the learning rate in gradient descent by proposing Meta-Regularization, which adds a regularization term on the learning rate and formulates it as a maxmin problem, resulting in algorithms that show comparable convergence and improved performance under strong-convexity in numerical experiments on benchmark problems.
We propose \textit{Meta-Regularization}, a novel approach for the adaptive choice of the learning rate in first-order gradient descent methods. Our approach modifies the objective function by adding a regularization term on the learning rate, and casts the joint updating process of parameters and learning rates into a maxmin problem. Given any regularization term, our approach facilitates the generation of practical algorithms. When \textit{Meta-Regularization} takes the $\varphi$-divergence as a regularizer, the resulting algorithms exhibit comparable theoretical convergence performance with other first-order gradient-based algorithms. Furthermore, we theoretically prove that some well-designed regularizers can improve the convergence performance under the strong-convexity condition of the objective function. Numerical experiments on benchmark problems demonstrate the effectiveness of algorithms derived from some common $\varphi$-divergence in full batch as well as online learning settings.