Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
This work addresses the need for more expressive priors in Bayesian neural networks, offering a way to incorporate function-space properties like periodicity, but it is incremental as it builds on existing Gaussian process and neural network methods.
The paper tackled the problem of independent weight priors in probabilistic neural networks by introducing hierarchical Gaussian process priors for weights, which capture correlations and allow prior knowledge inclusion, resulting in improved uncertainty estimates on out-of-distribution data and competitive predictive performance on an active learning benchmark.
Probabilistic neural networks are typically modeled with independent weight priors, which do not capture weight correlations in the prior and do not provide a parsimonious interface to express properties in function space. A desirable class of priors would represent weights compactly, capture correlations between weights, facilitate calibrated reasoning about uncertainty, and allow inclusion of prior knowledge about the function space such as periodicity or dependence on contexts such as inputs. To this end, this paper introduces two innovations: (i) a Gaussian process-based hierarchical model for network weights based on unit embeddings that can flexibly encode correlated weight structures, and (ii) input-dependent versions of these weight priors that can provide convenient ways to regularize the function space through the use of kernels defined on contextual inputs. We show these models provide desirable test-time uncertainty estimates on out-of-distribution data, demonstrate cases of modeling inductive biases for neural networks with kernels which help both interpolation and extrapolation from training data, and demonstrate competitive predictive performance on an active learning benchmark.