Dynamic Estimation of Learning Rates Using a Non-Linear Autoregressive Model
This work addresses optimization challenges in machine learning, particularly for adaptive learning rate tuning, but appears incremental as it builds on existing momentum and clipping concepts.
The paper tackles the problem of dynamically estimating learning rates and momentum in optimization by introducing a new class of adaptive non-linear autoregressive models, with results showing robust convergence and rapid adaptability across datasets and a reinforcement learning environment.
We introduce a new class of adaptive non-linear autoregressive (Nlar) models incorporating the concept of momentum, which dynamically estimate both the learning rates and momentum as the number of iterations increases. In our method, the growth of the gradients is controlled using a scaling (clipping) function, leading to stable convergence. Within this framework, we propose three distinct estimators for learning rates and provide theoretical proof of their convergence. We further demonstrate how these estimators underpin the development of effective Nlar optimizers. The performance of the proposed estimators and optimizers is rigorously evaluated through extensive experiments across several datasets and a reinforcement learning environment. The results highlight two key features of the Nlar optimizers: robust convergence despite variations in underlying parameters, including large initial learning rates, and strong adaptability with rapid convergence during the initial epochs.