Large-width functional asymptotics for deep Gaussian neural networks
This work provides foundational theoretical insights for researchers in machine learning and statistics, though it is incremental as it extends previous results on neural network-Gaussian process connections.
The paper tackles the theoretical analysis of deep Gaussian neural networks in the large-width limit, showing that they converge weakly to continuous Gaussian processes with almost surely locally Hölder continuous paths, establishing weak convergence in function-space with respect to a stronger metric.
In this paper, we consider fully connected feed-forward deep neural networks where weights and biases are independent and identically distributed according to Gaussian distributions. Extending previous results (Matthews et al., 2018a;b; Yang, 2019) we adopt a function-space perspective, i.e. we look at neural networks as infinite-dimensional random elements on the input space $\mathbb{R}^I$. Under suitable assumptions on the activation function we show that: i) a network defines a continuous Gaussian process on the input space $\mathbb{R}^I$; ii) a network with re-scaled weights converges weakly to a continuous Gaussian process in the large-width limit; iii) the limiting Gaussian process has almost surely locally $γ$-Hölder continuous paths, for $0 < γ<1$. Our results contribute to recent theoretical studies on the interplay between infinitely wide deep neural networks and Gaussian processes by establishing weak convergence in function-space with respect to a stronger metric.