Wilsonian Renormalization of Neural Network Gaussian Processes
This work provides a novel theoretical framework for understanding feature learning in deep neural networks, potentially identifying universality classes, but it is incremental as it builds on existing analogies between RG and neural networks.
The authors tackled the problem of separating relevant and irrelevant information in Gaussian Process (GP) regression by applying Wilsonian renormalization group (RG) methods, resulting in an analytically tractable RG flow that connects RG to learnable vs. unlearnable modes in neural networks.
Separating relevant and irrelevant information is key to any modeling process or scientific inquiry. Theoretical physics offers a powerful tool for achieving this in the form of the renormalization group (RG). Here we demonstrate a practical approach to performing Wilsonian RG in the context of Gaussian Process (GP) Regression. We systematically integrate out the unlearnable modes of the GP kernel, thereby obtaining an RG flow of the GP in which the data sets the IR scale. In simple cases, this results in a universal flow of the ridge parameter, which becomes input-dependent in the richer scenario in which non-Gaussianities are included. In addition to being analytically tractable, this approach goes beyond structural analogies between RG and neural networks by providing a natural connection between RG flow and learnable vs. unlearnable modes. Studying such flows may improve our understanding of feature learning in deep neural networks, and enable us to identify potential universality classes in these models.