Can a Confident Prior Replace a Cold Posterior?
This addresses the challenge of interpreting Bayesian methods in deep learning, offering a potential alternative to tempering for practitioners, though it is incremental as it builds on existing solutions.
The paper tackles the problem of Bayesian neural networks underfitting on low-noise image classification datasets by exploring whether a confidence-inducing prior can replace posterior tempering, finding that a 'DirClip' prior nearly matches cold posterior performance.
Benchmark datasets used for image classification tend to have very low levels of label noise. When Bayesian neural networks are trained on these datasets, they often underfit, misrepresenting the aleatoric uncertainty of the data. A common solution is to cool the posterior, which improves fit to the training data but is challenging to interpret from a Bayesian perspective. We explore whether posterior tempering can be replaced by a confidence-inducing prior distribution. First, we introduce a "DirClip" prior that is practical to sample and nearly matches the performance of a cold posterior. Second, we introduce a "confidence prior" that directly approximates a cold likelihood in the limit of decreasing temperature but cannot be easily sampled. Lastly, we provide several general insights into confidence-inducing priors, such as when they might diverge and how fine-tuning can mitigate numerical instability.