More Compute Is What You Need
This addresses the compute allocation problem for large language model practitioners, offering a unified scaling law that could optimize training strategies, though it appears incremental as it builds on existing scaling law concepts.
The paper hypothesizes a new scaling law suggesting that transformer-based model performance depends primarily on compute, independent of model and dataset size allocation, predicting that for inference efficiency, training should prioritize smaller models and larger datasets, and that scaling model size may be the only way to improve performance if web datasets are exhausted.
Large language model pre-training has become increasingly expensive, with most practitioners relying on scaling laws to allocate compute budgets for model size and training tokens, commonly referred to as Compute-Optimal or Chinchilla Optimal. In this paper, we hypothesize a new scaling law that suggests model performance depends mostly on the amount of compute spent for transformer-based models, independent of the specific allocation to model size and dataset size. Using this unified scaling law, we predict that (a) for inference efficiency, training should prioritize smaller model sizes and larger training datasets, and (b) assuming the exhaustion of available web datasets, scaling the model size might be the only way to further improve model performance.