Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence
This addresses the need for better default schedulers in machine learning training, offering a computationally efficient solution that is incremental over existing methods like Cosine decay.
The paper tackles the problem of inefficient learning rate scheduling in training by proposing GreedyLR, a scheduler that adapts based on loss changes, resulting in faster convergence and improved accuracy across NLP, CV, and LLM tasks with up to 7B parameters.
Despite significant advances in optimizers for training, most research works use common scheduler choices like Cosine or exponential decay. In this paper, we study \emph{GreedyLR}, a novel scheduler that adaptively adjusts the learning rate during training based on the current loss. To validate the effectiveness of our proposed scheduler, we conduct experiments on several NLP, CV, and LLM tasks with up to $7B$ parameters, including both fine-tuning and pre-training experiments. The results show that our approach outperforms several state-of-the-art schedulers in terms of accuracy, speed, and convergence. We also provide a theoretical analysis of the GreedyLR algorithm, including a proof of convergence and derivation of the optimal scaling factor $F$ that maximizes the convergence rate, along with experiments to show robustness of the algorithm to realistic noisy landscapes. Our scheduler is easy to implement, computationally efficient, and could be considered a good default scheduler for training.