Predictive Complexity Priors
This addresses the problem of unintuitive prior effects in Bayesian modeling for researchers and practitioners, though it appears incremental as it builds on existing functional prior concepts.
The authors tackled the difficulty of specifying Bayesian priors for complex models like neural networks by proposing predictive complexity priors, which are defined by comparing model predictions to a reference model and transferred to parameters, resulting in applications in high-dimensional regression, neural network depth reasoning, and few-shot learning.
Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model's predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model's predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.