Smoothness-Based Derandomization of PAC-Bayes Bounds
For machine learning theorists and practitioners, this work provides a principled way to obtain high-probability generalization bounds for deterministic models (e.g., neural networks) by connecting smoothness to PAC-Bayes derandomization, though the experiments are limited to illustrating the regularizer's behavior rather than demonstrating strong empirical gains.
The paper derives PAC-Bayes generalization bounds for deterministic predictors by leveraging smoothness of the loss and predictor class, quantifying the cost of derandomization via the Jensen gap class and its Rademacher complexity. The bounds involve flatness measures (Jacobians and Hessians) and motivate a practical regularizer, with experiments on CIFAR-10 showing its behavior under different batch sizes.
We study PAC-Bayes derandomization for smooth loss functions. Our goal is to obtain generalization bounds that hold with high probability for deterministic predictors by exploiting smoothness properties of both the loss and the predictor class. We show that passing from the Gibbs predictor to the deterministic predictor at the posterior mean has a precise cost, given by the generalization gap of the Jensen gap class. We control this class through its Rademacher complexity, leading to bounds for deterministic predictors that involve flatness quantities expressed in terms of parameter Jacobians and Hessians of the score map. The framework applies to both bounded and unbounded smooth loss functions, and we specialize the results to linear predictors and smooth neural networks. Finally, the Jacobian and Hessian quantities appearing in the theory motivate a practical regularizer. For BatchNorm networks, we compute this regularizer with respect to effective BatchNorm weights obtained by folding the BatchNorm transformation into the adjacent affine weights. Experiments on CIFAR-10 illustrate the behavior of this regularizer under different batch sizes.