The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks
This work provides theoretical insights for researchers in machine learning, particularly in understanding generalization in kernel methods and wide neural networks, though it is incremental in building on prior conservation law concepts.
The authors derived simple closed-form estimates for test risk in kernel ridge regression by identifying a conservation law, enabling clearer interpretations and applications such as explaining the 'deep bootstrap' and linking to statistical physics.
We derive simple closed-form estimates for the test risk and other generalization metrics of kernel ridge regression (KRR). Relative to prior work, our derivations are greatly simplified and our final expressions are more readily interpreted. These improvements are enabled by our identification of a sharp conservation law which limits the ability of KRR to learn any orthonormal basis of functions. Test risk and other objects of interest are expressed transparently in terms of our conserved quantity evaluated in the kernel eigenbasis. We use our improved framework to: i) provide a theoretical explanation for the "deep bootstrap" of Nakkiran et al (2020), ii) generalize a previous result regarding the hardness of the classic parity problem, iii) fashion a theoretical tool for the study of adversarial robustness, and iv) draw a tight analogy between KRR and a well-studied system in statistical physics.