Excess risk analysis for epistemic uncertainty with application to variational inference
This work addresses the lack of theoretical analysis for epistemic uncertainty in Bayesian deep learning, offering a method to enhance uncertainty evaluation in supervised learning tasks.
The authors analyzed epistemic uncertainty in approximate Bayesian inference by linking generalization error to common uncertainty measures and derived convergence behaviors. They proposed a new variational inference objective that directly controls prediction performance and epistemic uncertainty, showing significant improvement over existing methods in numerical experiments.
Bayesian deep learning plays an important role especially for its ability evaluating epistemic uncertainty (EU). Due to computational complexity issues, approximation methods such as variational inference (VI) have been used in practice to obtain posterior distributions and their generalization abilities have been analyzed extensively, for example, by PAC-Bayesian theory; however, little analysis exists on EU, although many numerical experiments have been conducted on it. In this study, we analyze the EU of supervised learning in approximate Bayesian inference by focusing on its excess risk. First, we theoretically show the novel relations between generalization error and the widely used EU measurements, such as the variance and mutual information of predictive distribution, and derive their convergence behaviors. Next, we clarify how the objective function of VI regularizes the EU. With this analysis, we propose a new objective function for VI that directly controls the prediction performance and the EU based on the PAC-Bayesian theory. Numerical experiments show that our algorithm significantly improves the EU evaluation over the existing VI methods.