On the generalization of bayesian deep nets for multi-class classification
This work addresses the challenge of improving generalization guarantees for Bayesian deep learning in multi-class classification, though it appears incremental as it builds on existing bound techniques with a new term.
The authors tackled the problem of deriving generalization bounds for Bayesian deep networks without relying on strict assumptions like uniformly bounded or Lipschitz loss functions, by proposing a new bound that incorporates a loss-gradient norm term via Log-Sobolev inequalities, and empirically analyzed this term across various deep networks.
Generalization bounds which assess the difference between the true risk and the empirical risk have been studied extensively. However, to obtain bounds, current techniques use strict assumptions such as a uniformly bounded or a Lipschitz loss function. To avoid these assumptions, in this paper, we propose a new generalization bound for Bayesian deep nets by exploiting the contractivity of the Log-Sobolev inequalities. Using these inequalities adds an additional loss-gradient norm term to the generalization bound, which is intuitively a surrogate of the model complexity. Empirically, we analyze the affect of this loss-gradient norm term using different deep nets.