The lazy (NTK) and rich ($μ$P) regimes: a gentle tutorial
It synthesizes recent research to provide insights into feature learning in deep neural networks, which is incremental as it builds on existing theoretical work.
This tutorial explains that for training wide neural networks, hyperparameters like learning rate and initial weight size reduce to a single degree of freedom controlling training behavior, ranging from lazy kernel-like training to active feature learning in the μP regime.
A central theme of the modern machine learning paradigm is that larger neural networks achieve better performance on a variety of metrics. Theoretical analyses of these overparameterized models have recently centered around studying very wide neural networks. In this tutorial, we provide a nonrigorous but illustrative derivation of the following fact: in order to train wide networks effectively, there is only one degree of freedom in choosing hyperparameters such as the learning rate and the size of the initial weights. This degree of freedom controls the richness of training behavior: at minimum, the wide network trains lazily like a kernel machine, and at maximum, it exhibits feature learning in the active $μ$P regime. In this paper, we explain this richness scale, synthesize recent research results into a coherent whole, offer new perspectives and intuitions, and provide empirical evidence supporting our claims. In doing so, we hope to encourage further study of the richness scale, as it may be key to developing a scientific theory of feature learning in practical deep neural networks.