Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weights
This provides foundational theoretical support for widely used BNNs with Gaussian priors, addressing a key gap in the literature.
The paper tackles the lack of theoretical results for Bayesian neural networks (BNNs) with Gaussian priors by developing a new approximation theory for non-sparse deep neural networks with bounded parameters, and shows that BNNs with general priors achieve near-minimax optimal posterior concentration rates.
Bayesian approaches for training deep neural networks (BNNs) have received significant interest and have been effectively utilized in a wide range of applications. There have been several studies on the properties of posterior concentrations of BNNs. However, most of these studies only demonstrate results in BNN models with sparse or heavy-tailed priors. Surprisingly, no theoretical results currently exist for BNNs using Gaussian priors, which are the most commonly used one. The lack of theory arises from the absence of approximation results of Deep Neural Networks (DNNs) that are non-sparse and have bounded parameters. In this paper, we present a new approximation theory for non-sparse DNNs with bounded parameters. Additionally, based on the approximation theory, we show that BNNs with non-sparse general priors can achieve near-minimax optimal posterior concentration rates to the true model.