eagle: early approximated gradient based learning rate estimator
This addresses training efficiency for deep learning practitioners, though it appears incremental as it builds on existing optimizers like Adam with an adaptive switching mechanism.
The paper tackles the problem of slow training convergence in deep learning by proposing EAGLE, an optimization method that accelerates loss convergence during early training stages using parameter and gradient changes from consecutive steps. Experiments show EAGLE achieves faster training loss convergence with fewer epochs compared to conventional methods.
We propose EAGLE update rule, a novel optimization method that accelerates loss convergence during the early stages of training by leveraging both current and previous step parameter and gradient values. The update algorithm estimates optimal parameters by computing the changes in parameters and gradients between consecutive training steps and leveraging the local curvature of the loss landscape derived from these changes. However, this update rule has potential instability, and to address that, we introduce an adaptive switching mechanism that dynamically selects between Adam and EAGLE update rules to enhance training stability. Experiments on standard benchmark datasets demonstrate that EAGLE optimizer, which combines this novel update rule with the switching mechanism achieves rapid training loss convergence with fewer epochs, compared to conventional optimization methods.