Six Lectures on Linearized Neural Networks
Theodor Misiakiewicz Andrea Montanari
In these six lectures, we examine what can be learnt about the behavior of
multi-layer neural networks from the analysis of linear models. We first recall
the correspondence between neural networks and linear models via the so-called
lazy regime. We then review four models for linearized neural networks: linear
regression with concentrated features, kernel ridge regression, random feature
model and neural tangent model. Finally, we highlight the limitations of the
linear theory and discuss how other approaches can overcome them.