Feature Preserving Shrinkage on Bayesian Neural Networks via the R2D2 Prior
This work addresses a critical bottleneck in BNNs for researchers and practitioners by improving prior selection to enhance model reliability, though it is incremental as it builds on existing BNN frameworks.
The paper tackles the challenge of selecting appropriate prior distributions in Bayesian neural networks (BNNs), which can lead to poor predictive performance or inflated variance, by proposing the R2D2-Net that imposes an R2D2 prior to effectively shrink irrelevant coefficients while preserving key features, resulting in satisfactory performance on image classification and uncertainty estimation tasks.
Bayesian neural networks (BNNs) treat neural network weights as random variables, which aim to provide posterior uncertainty estimates and avoid overfitting by performing inference on the posterior weights. However, the selection of appropriate prior distributions remains a challenging task, and BNNs may suffer from catastrophic inflated variance or poor predictive performance when poor choices are made for the priors. Existing BNN designs apply different priors to weights, while the behaviours of these priors make it difficult to sufficiently shrink noisy signals or they are prone to overshrinking important signals in the weights. To alleviate this problem, we propose a novel R2D2-Net, which imposes the R^2-induced Dirichlet Decomposition (R2D2) prior to the BNN weights. The R2D2-Net can effectively shrink irrelevant coefficients towards zero, while preventing key features from over-shrinkage. To approximate the posterior distribution of weights more accurately, we further propose a variational Gibbs inference algorithm that combines the Gibbs updating procedure and gradient-based optimization. This strategy enhances stability and consistency in estimation when the variational objective involving the shrinkage parameters is non-convex. We also analyze the evidence lower bound (ELBO) and the posterior concentration rates from a theoretical perspective. Experiments on both natural and medical image classification and uncertainty estimation tasks demonstrate satisfactory performance of our method.