PAC-Bayes-Chernoff bounds for unbounded losses
This work addresses a foundational challenge in machine learning theory for researchers, offering incremental improvements by extending existing bounds to unbounded losses with richer assumptions.
The paper tackles the problem of deriving generalization bounds for unbounded losses in PAC-Bayesian learning by introducing a new PAC-Bayes oracle bound based on Cramér-Chernoff bounds, resulting in more informative and potentially tighter bounds that generalize previous results and provide theoretical coverage for regularization techniques.
We introduce a new PAC-Bayes oracle bound for unbounded losses that extends Cramér-Chernoff bounds to the PAC-Bayesian setting. The proof technique relies on controlling the tails of certain random variables involving the Cramér transform of the loss. Our approach naturally leverages properties of Cramér-Chernoff bounds, such as exact optimization of the free parameter in many PAC-Bayes bounds. We highlight several applications of the main theorem. Firstly, we show that our bound recovers and generalizes previous results. Additionally, our approach allows working with richer assumptions that result in more informative and potentially tighter bounds. In this direction, we provide a general bound under a new \textit{model-dependent} assumption from which we obtain bounds based on parameter norms and log-Sobolev inequalities. Notably, many of these bounds can be minimized to obtain distributions beyond the Gibbs posterior and provide novel theoretical coverage to existing regularization techniques.