Approximate Inference Turns Deep Networks into Gaussian Processes
This work provides a theoretical connection that could facilitate further research on combining DNNs and GPs in practical settings, though it appears incremental in linking existing concepts.
The paper tackles the problem of understanding the relationship between training methods for deep neural networks (DNNs) and Gaussian processes (GPs) by showing that certain Gaussian posterior approximations for Bayesian DNNs are equivalent to GP posteriors, resulting in a kernel that is the neural tangent kernel and enabling hyperparameter tuning via GP marginal likelihood.
Deep neural networks (DNN) and Gaussian processes (GP) are two powerful models with several theoretical connections relating them, but the relationship between their training methods is not well understood. In this paper, we show that certain Gaussian posterior approximations for Bayesian DNNs are equivalent to GP posteriors. This enables us to relate solutions and iterations of a deep-learning algorithm to GP inference. As a result, we can obtain a GP kernel and a nonlinear feature map while training a DNN. Surprisingly, the resulting kernel is the neural tangent kernel. We show kernels obtained on real datasets and demonstrate the use of the GP marginal likelihood to tune hyperparameters of DNNs. Our work aims to facilitate further research on combining DNNs and GPs in practical settings.