Benchmarking Optimizers for Large Language Model Pretraining
This work addresses the problem of inconsistent benchmarking for researchers and practitioners in machine learning, offering a reproducible framework, but it is incremental as it focuses on evaluation rather than introducing new methods.
The study tackled the challenge of comparing diverse optimization methods for large language model pretraining by conducting a comprehensive evaluation across standardized scenarios, providing guidance on optimizer selection for practitioners and highlighting future research directions.
The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Claims from those methods are myriad: from faster convergence to removing reliance on certain hyperparameters. However, the diverse experimental protocols used to validate these claims make direct comparisons between methods challenging. This study presents a comprehensive evaluation of recent optimization techniques across standardized LLM pretraining scenarios, systematically varying model size, batch size, and training duration. Through careful tuning of each method, we provide guidance to practitioners on which optimizer is best suited for each scenario. For researchers, our work highlights promising directions for future optimization research. Finally, by releasing our code and making all experiments fully reproducible, we hope our efforts can help the development and rigorous benchmarking of future methods.