Proportional infinite-width infinite-depth limit for deep linear neural networks
This work addresses a foundational problem in deep learning theory for researchers, providing a more descriptive limit for neural networks, though it is incremental as it builds on prior infinite-width results and focuses on linear activations.
The paper tackles the limitation of the infinite-width Gaussian process limit for neural networks, which lacks descriptive power for learning dependent features and output correlations, by exploring a joint proportional limit where both depth and width diverge with a constant ratio, resulting in a non-Gaussian distribution that retains correlations between outputs, specifically characterizing it as a nontrivial mixture of Gaussians for linear activation functions.
We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. Prior works have shown that in the infinite-width regime, where the number of neurons per layer grows to infinity while the depth remains fixed, neural networks converge to a Gaussian process, known as the Neural Network Gaussian Process. However, this Gaussian limit sacrifices descriptive power, as it lacks the ability to learn dependent features and produce output correlations that reflect observed labels. Motivated by these limitations, we explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs. Our contribution extends previous works by rigorously characterizing, for linear activation functions, the limiting distribution as a nontrivial mixture of Gaussians.