On change of measure inequalities for $f$-divergences
This work provides theoretical tools for machine learning generalization analysis, particularly in non-standard scenarios, but it is incremental as it extends existing divergence-based methods.
The authors tackled the problem of deriving change of measure inequalities for f-divergences, resulting in new PAC-Bayesian generalization bounds that apply to settings like heavy-tailed losses, with instantiations for popular f-divergences.
We propose new change of measure inequalities based on $f$-divergences (of which the Kullback-Leibler divergence is a particular case). Our strategy relies on combining the Legendre transform of $f$-divergences and the Young-Fenchel inequality. By exploiting these new change of measure inequalities, we derive new PAC-Bayesian generalisation bounds with a complexity involving $f$-divergences, and holding in mostly unchartered settings (such as heavy-tailed losses). We instantiate our results for the most popular $f$-divergences.