Non-Gaussian processes and neural networks at finite widths
This work provides a theoretical framework for understanding finite-width neural networks, which is incremental but addresses a known limitation in neural network theory.
The authors tackled the problem of extending the correspondence between neural networks and Gaussian processes from infinite-width to finite-width networks, resulting in non-Gaussian processes as priors and a perturbative method for Bayesian inference with these priors.
Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow. We further develop a perturbative procedure to perform Bayesian inference with weakly non-Gaussian priors.