Six Lectures on Linearized Neural Networks
This provides a theoretical framework for researchers in machine learning, but it is incremental as it reviews existing concepts.
The lectures explore how linear models can inform the understanding of multi-layer neural networks by reviewing four linearized models and discussing their limitations.
In these six lectures, we examine what can be learnt about the behavior of multi-layer neural networks from the analysis of linear models. We first recall the correspondence between neural networks and linear models via the so-called lazy regime. We then review four models for linearized neural networks: linear regression with concentrated features, kernel ridge regression, random feature model and neural tangent model. Finally, we highlight the limitations of the linear theory and discuss how other approaches can overcome them.