A brief note on understanding neural networks as Gaussian processes
This provides a theoretical interpretation linking neural networks to Gaussian processes, which is incremental as it builds on existing research.
The paper generalizes prior work to analyze when neural network outputs follow Gaussian processes, showing that posterior mean functions lie in reproducing kernel Hilbert spaces defined by neural-network-induced kernels, and for two-layer networks, these spaces form a Barron space.
As a generalization of the work in [Lee et al., 2017], this note briefly discusses when the prior of a neural network output follows a Gaussian process, and how a neural-network-induced Gaussian process is formulated. The posterior mean functions of such a Gaussian process regression lie in the reproducing kernel Hilbert space defined by the neural-network-induced kernel. In the case of two-layer neural networks, the induced Gaussian processes provide an interpretation of the reproducing kernel Hilbert spaces whose union forms a Barron space.