Improving Line Search Methods for Large Scale Neural Network Training
This work addresses optimization challenges for researchers and practitioners training large-scale neural networks, though it is incremental as it builds on existing line search methods.
The paper tackled issues in state-of-the-art line search methods for neural network training by enhancing the Armijo line search with momentum from ADAM, resulting in improved performance over previous implementations and tuned learning rate schedules for Adam on Transformers and CNNs in NLP and image data.
In recent studies, line search methods have shown significant improvements in the performance of traditional stochastic gradient descent techniques, eliminating the need for a specific learning rate schedule. In this paper, we identify existing issues in state-of-the-art line search methods, propose enhancements, and rigorously evaluate their effectiveness. We test these methods on larger datasets and more complex data domains than before. Specifically, we improve the Armijo line search by integrating the momentum term from ADAM in its search direction, enabling efficient large-scale training, a task that was previously prone to failure using Armijo line search methods. Our optimization approach outperforms both the previous Armijo implementation and tuned learning rate schedules for Adam. Our evaluation focuses on Transformers and CNNs in the domains of NLP and image data. Our work is publicly available as a Python package, which provides a hyperparameter free Pytorch optimizer.