VIB is Half Bayes
This work provides an incremental understanding of the theoretical underpinnings of VIB for machine learning researchers and practitioners.
This paper demonstrates that the Variational Information Bottleneck (VIB) acts as a compromise between empirical and fully Bayesian objectives in discriminative tasks. It specifically minimizes risks arising from finite sampling of the target variable Y.
In discriminative settings such as regression and classification there are two random variables at play, the inputs X and the targets Y. Here, we demonstrate that the Variational Information Bottleneck can be viewed as a compromise between fully empirical and fully Bayesian objectives, attempting to minimize the risks due to finite sampling of Y only. We argue that this approach provides some of the benefits of Bayes while requiring only some of the work.