Faster Convergence for Transformer Fine-tuning with Line Search Methods
This work addresses convergence issues in fine-tuning Transformers, particularly beneficial for resource-limited scenarios, though it is incremental as it adapts existing line search methods to a new architecture.
The authors tackled the problem of slow convergence in Transformer fine-tuning by extending line search methods to the Transformer architecture, achieving significant performance improvements for small datasets or training budgets, with equal or better results in other cases.
Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units and perform the line search separately on these local units. Our optimization method outperforms the traditional Adam optimizer and achieves significant performance improvements for small data sets or small training budgets, while performing equal or better for other tested cases. Our work is publicly available as a python package, which provides a hyperparameter-free pytorch optimizer that is compatible with arbitrary network architectures.