On the Importance of Gradient Norm in PAC-Bayesian Bounds
This work addresses the need for more flexible generalization bounds in machine learning, particularly for Bayesian deep learning, but it is incremental as it builds on existing PAC-Bayesian and log-Sobolev inequality frameworks.
The paper tackles the problem of deriving generalization bounds without strict assumptions like uniformly bounded or Lipschitz loss functions, by relaxing these to on-average bounded loss and gradient norm assumptions, resulting in a new bound that incorporates a loss-gradient norm term as a surrogate for model complexity, with empirical analysis on Bayesian deep nets showing its effect across neural architectures.
Generalization bounds which assess the difference between the true risk and the empirical risk, have been studied extensively. However, to obtain bounds, current techniques use strict assumptions such as a uniformly bounded or a Lipschitz loss function. To avoid these assumptions, in this paper, we follow an alternative approach: we relax uniform bounds assumptions by using on-average bounded loss and on-average bounded gradient norm assumptions. Following this relaxation, we propose a new generalization bound that exploits the contractivity of the log-Sobolev inequalities. These inequalities add an additional loss-gradient norm term to the generalization bound, which is intuitively a surrogate of the model complexity. We apply the proposed bound on Bayesian deep nets and empirically analyze the effect of this new loss-gradient norm term on different neural architectures.