Hi-fi functional priors by learning activations
This addresses regularization and uncertainty quantification in BNNs for machine learning practitioners, but appears incremental as it builds on existing function-space prior methods.
The paper tackled the challenge of imposing function-space priors in Bayesian Neural Networks (BNNs) by using trainable activations, and found that even BNNs with a single wide hidden layer could effectively achieve desired priors.
Function-space priors in Bayesian Neural Networks (BNNs) provide a more intuitive approach to embedding beliefs directly into the model's output, thereby enhancing regularization, uncertainty quantification, and risk-aware decision-making. However, imposing function-space priors on BNNs is challenging. We address this task through optimization techniques that explore how trainable activations can accommodate higher-complexity priors and match intricate target function distributions. We investigate flexible activation models, including Pade functions and piecewise linear functions, and discuss the learning challenges related to identifiability, loss construction, and symmetries. Our empirical findings indicate that even BNNs with a single wide hidden layer when equipped with flexible trainable activation, can effectively achieve desired function-space priors.