On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors
This work addresses the impracticality of BNN posteriors for inference, which is a problem for researchers in Bayesian deep learning, but it appears incremental as it builds on prior studies of overparametrization.
The authors tackled the problem of understanding how overparametrization and priors reshape Bayesian neural network posteriors, showing that redundancy introduces phenomena like balancedness and prior conformity, and validated this with extensive experiments.
Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.