Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling
This work addresses foundational questions about guiding principles and model comparison for researchers and practitioners in machine learning, particularly those working with large-scale models, though it is incremental in re-evaluating existing ideas.
The paper examines whether established regularization principles from the generalization-centric era remain valid in the scaling-centric era of large language models, finding that methods like L2 regularization and small batch sizes may not hold at larger scales and identifying a 'scaling law crossover' phenomenon.
The remarkable success of large language pretraining and the discovery of scaling laws signify a paradigm shift in machine learning. Notably, the primary objective has evolved from minimizing generalization error to reducing approximation error, and the most effective strategy has transitioned from regularization (in a broad sense) to scaling up models. This raises a critical question: Do the established principles that proved successful in the generalization-centric era remain valid in this new era of scaling? This paper examines several influential regularization-based principles that may no longer hold true in the scaling-centric, large language model (LLM) era. These principles include explicit L2 regularization and implicit regularization through small batch sizes and large learning rates. Additionally, we identify a new phenomenon termed ``scaling law crossover,'' where two scaling curves intersect at a certain scale, implying that methods effective at smaller scales may not generalize to larger ones. Together, these observations highlight two fundamental questions within this new paradigm: $\bullet$ Guiding Principles for Scaling: If regularization is no longer the primary guiding principle for model design, what new principles are emerging to guide scaling? $\bullet$ Model Comparison at Scale: How to reliably and effectively compare models at the scale where only a single experiment is feasible?