Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
This work provides practical scaling formulas for hyperparameter tuning in LLM pre-training, addressing efficiency challenges for researchers and engineers, though it is incremental as it builds on prior scaling law research.
The authors derived scaling laws for hyperparameters in LLM pre-training, showing that optimal weight decay scales linearly with batch size and follows a power law in tokens-per-parameter ratio, enabling accurate prediction before large-scale training. They also found that optimal and critical batch sizes scale as power laws in dataset size, independent of model size, and applied these insights to Pareto-optimal model and dataset selection under training time and compute constraints.
Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate η and weight decay λ. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and batch size B. Recent work suggests the AdamW timescale, B/(ηλD), should remain constant across training settings, and we verify the implication that optimal λ scales linearly with B, for a fixed N,D. However, as N,D scale, we show the optimal timescale obeys a precise power law in the tokens-per-parameter ratio, D/N. This law thus provides a method to accurately predict λopt in advance of large-scale training. We also study scaling laws for optimal batch size Bopt (the B enabling lowest loss at a given N,D) and critical batch size Bcrit (the B beyond which further data parallelism becomes ineffective). In contrast with prior work, we find both Bopt and Bcrit scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and D under dual training time and compute objectives.