More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize
This work addresses the theoretical gap in understanding generalization for practitioners using large-scale neural networks, though it is incremental by applying existing estimators to new contexts.
The authors tackled the problem of why overparameterized models generalize in real-world settings, finding that the classical GCV estimator accurately predicts generalization risk for neural network kernels on real data like CIFAR-100, and used random matrix theory to explain factors like pretraining benefits and scaling laws.
Of theories for why large-scale machine learning models generalize despite being vastly overparameterized, which of their assumptions are needed to capture the qualitative phenomena of generalization in the real world? On one hand, we find that most theoretical analyses fall short of capturing these qualitative phenomena even for kernel regression, when applied to kernels derived from large-scale neural networks (e.g., ResNet-50) and real data (e.g., CIFAR-100). On the other hand, we find that the classical GCV estimator (Craven and Wahba, 1978) accurately predicts generalization risk even in such overparameterized settings. To bolster this empirical finding, we prove that the GCV estimator converges to the generalization risk whenever a local random matrix law holds. Finally, we apply this random matrix theory lens to explain why pretrained representations generalize better as well as what factors govern scaling laws for kernel regression. Our findings suggest that random matrix theory, rather than just being a toy model, may be central to understanding the properties of neural representations in practice.