No learning rates needed: Introducing SALSA -- Stable Armijo Line Search Adaptation
This work addresses the need for robust optimization methods without manual learning rate tuning, offering incremental improvements for machine learning practitioners using stochastic gradient descent.
The paper tackled the problem of improving line search methods for stochastic gradient descent by enhancing the Armijo line search with faster computation and a momentum term, resulting in an optimization approach that outperforms previous implementations and tuned learning rate schedules for Adam and SGD across various architectures and data domains.
In recent studies, line search methods have been demonstrated to significantly enhance the performance of conventional stochastic gradient descent techniques across various datasets and architectures, while making an otherwise critical choice of learning rate schedule superfluous. In this paper, we identify problems of current state-of-the-art of line search methods, propose enhancements, and rigorously assess their effectiveness. Furthermore, we evaluate these methods on orders of magnitude larger datasets and more complex data domains than previously done. More specifically, we enhance the Armijo line search method by speeding up its computation and incorporating a momentum term into the Armijo criterion, making it better suited for stochastic mini-batching. Our optimization approach outperforms both the previous Armijo implementation and a tuned learning rate schedule for the Adam and SGD optimizers. Our evaluation covers a diverse range of architectures, such as Transformers, CNNs, and MLPs, as well as data domains, including NLP and image data. Our work is publicly available as a Python package, which provides a simple Pytorch optimizer.