Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle
Provides theoretical justification for the Gaussian process behavior of wide neural networks, relevant for understanding generalization and initialization in deep learning.
The paper proves quantitative bounds on the 2-Wasserstein distance between deep neural networks and their infinite-width Gaussian limit using a Lindeberg exchange principle, establishing universality for general weights and activation functions.
We consider the infinite-width limit of a fully connected deep neural network with general weights, and we prove quantitative general bounds on the $2$-Wasserstein distance between the network and its infinite-width Gaussian limit, under appropriate regularity assumptions on the activation function. Our main tool is a Lindeberg principle for Deep Neural Networks, which we use to successively replace the weights on each layer by Gaussian random variables.