A Constructive Prediction of the Generalization Error Across Scales
This work addresses a critical issue for machine learning practitioners and theorists by offering a predictive tool for generalization error, though it appears incremental as it builds on existing scaling concepts.
The authors tackled the problem of predicting neural network generalization error across model and dataset sizes by proposing a functional form that approximates this dependency well in practice, showing it fits observations and provides accurate predictions from small to large scales.
The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.