Simpler PAC-Bayesian Bounds for Hostile Data
This work addresses the need for more robust theoretical guarantees in machine learning for scenarios with non-standard data, though it appears incremental as it extends existing bounds rather than introducing a new paradigm.
The paper tackles the problem of deriving PAC-Bayesian learning bounds under relaxed assumptions, such as for dependent and heavy-tailed data, by replacing the Kullback-Leibler divergence with a general f-divergence, resulting in bounds applicable to hostile data settings.
PAC-Bayesian learning bounds are of the utmost interest to the learning community. Their role is to connect the generalization ability of an aggregation distribution $ρ$ to its empirical risk and to its Kullback-Leibler divergence with respect to some prior distribution $π$. Unfortunately, most of the available bounds typically rely on heavy assumptions such as boundedness and independence of the observations. This paper aims at relaxing these constraints and provides PAC-Bayesian learning bounds that hold for dependent, heavy-tailed observations (hereafter referred to as \emph{hostile data}). In these bounds the Kullack-Leibler divergence is replaced with a general version of Csiszár's $f$-divergence. We prove a general PAC-Bayesian bound, and show how to use it in various hostile settings.