Random Neural Networks in the Infinite Width Limit as Gaussian Processes
This provides a theoretical foundation for understanding neural network behavior in the infinite width limit, which is incremental but extends prior work by relaxing assumptions.
The paper tackles the problem of proving convergence of fully connected neural networks with random weights and biases to Gaussian processes as hidden layer widths approach infinity, showing this under only moment conditions and for general non-linearities.
This article gives a new proof that fully connected neural networks with random weights and biases converge to Gaussian processes in the regime where the input dimension, output dimension, and depth are kept fixed, while the hidden layer widths tend to infinity. Unlike prior work, convergence is shown assuming only moment conditions for the distribution of weights and for quite general non-linearities.