Checkpoint Merging via Bayesian Optimization in LLM Pretraining
This work addresses resource challenges in LLM pretraining for AI researchers and practitioners, but it is incremental as it builds on existing checkpoint merging techniques with a novel optimization approach.
The paper tackles the high computational and environmental costs of training large language models by proposing checkpoint merging during pretraining, using Bayesian optimization to find optimal merging weights, which demonstrates robust generalization across domains and offers substantial benefits at minimal cost.
The rapid proliferation of large language models (LLMs) such as GPT-4 and Gemini underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs. To alleviate this issue, we propose checkpoint merging in pretraining LLM. This method utilizes LLM checkpoints with shared training trajectories, and is rooted in an extensive search space exploration for the best merging weight via Bayesian optimization. Through various experiments, we demonstrate that: (1) Our proposed methodology exhibits the capacity to augment pretraining, presenting an opportunity akin to obtaining substantial benefits at minimal cost; (2) Our proposed methodology, despite requiring a given held-out dataset, still demonstrates robust generalization capabilities across diverse domains, a pivotal aspect in pretraining.