Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
This addresses the practical deployment issue of balancing training and inference costs for LLM researchers and practitioners, offering an incremental improvement over Chinchilla scaling laws.
The paper tackles the problem that existing language model scaling laws, like Chinchilla, ignore inference costs, and modifies them to optimize parameter count and data size for given quality and inference demand, finding that models should be trained smaller and longer for high inference demand (~1B requests) and validating this with 47 models showing quality improvements at extreme token/parameter ratios up to 10,000.
Large language model (LLM) scaling laws are empirical formulas that estimate changes in model quality as a result of increasing parameter count and training data. However, these formulas, including the popular Deepmind Chinchilla scaling laws, neglect to include the cost of inference. We modify the Chinchilla scaling laws to calculate the optimal LLM parameter count and pre-training data size to train and deploy a model of a given quality and inference demand. We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal. Furthermore, we train 47 models of varying sizes and parameter counts to validate our formula and find that model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000). Finally, we ablate the procedure used to fit the Chinchilla scaling law coefficients and find that developing scaling laws only from data collected at typical token/parameter ratios overestimates the impact of additional tokens at these extreme ranges.