Function-Space Regularization for Deep Bayesian Classification
This work addresses the challenge of model-specific and hard-to-interpret priors in Bayesian deep learning, offering a flexible approach for practitioners.
The paper tackles the problem of specifying priors in Bayesian deep learning by applying a Dirichlet prior in predictive space instead of weight space, enabling improved uncertainty quantification and adversarial robustness in image classification.
Bayesian deep learning approaches assume model parameters to be latent random variables and infer posterior distributions to quantify uncertainty, increase safety and trust, and prevent overconfident and unpredictable behavior. However, weight-space priors are model-specific, can be difficult to interpret and are hard to specify. Instead, we apply a Dirichlet prior in predictive space and perform approximate function-space variational inference. To this end, we interpret conventional categorical predictions from stochastic neural network classifiers as samples from an implicit Dirichlet distribution. By adapting the inference, the same function-space prior can be combined with different models without affecting model architecture or size. We illustrate the flexibility and efficacy of such a prior with toy experiments and demonstrate scalability, improved uncertainty quantification and adversarial robustness with large-scale image classification experiments.