A Hitchhiker's Guide to Scaling Law Estimation
This work addresses the need for reliable scaling law estimation to aid practitioners and researchers in making informed pretraining decisions, though it is incremental as it builds on existing scaling law concepts with new empirical insights.
The paper tackles the problem of accurately estimating scaling laws for machine learning models by analyzing a large dataset of 485 pretrained models to derive best practices, finding that using intermediate checkpoints and models of similar sizes improves accuracy, with variability across seeds making multiple small models sometimes more useful than a single large one.
Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets. This provides an efficient way for practitioners and researchers alike to compare pretraining decisions involving optimizers, datasets, and model architectures. Despite the widespread use of scaling laws to model the dynamics of language model training, there has been little work on understanding how to best estimate and interpret them. We collect (and release) a large-scale dataset containing losses and downstream evaluations for 485 previously published pretrained models. We use these to estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families. We find that fitting scaling laws to intermediate checkpoints of training runs (and not just their final losses) substantially improves accuracy, and that -- all else equal -- estimates of performance are generally most accurate when derived from other models of similar sizes. However, because there is a significant degree of variability across model seeds, training multiple small models is sometimes more useful than training a single large one. Moreover, while different model families differ scaling behavior, they are often similar enough that a target model's behavior can be predicted from a single model with the same architecture, along with scaling parameter estimates derived from other model families.