Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation
This work provides theoretical tools for improving statistical learning and estimation, but it is incremental as it builds on existing divergence frameworks.
The paper tackles the problem of deriving change of measure inequalities for f-divergences and α-divergences, resulting in novel bounds such as a multiplicative inequality for α-divergences and a generalized Hammersley-Chapman-Robbins inequality, with applications including PAC-Bayesian bounds for various loss classes and non-asymptotic intervals for Monte Carlo estimates.
We introduce several novel change of measure inequalities for two families of divergences: $f$-divergences and $α$-divergences. We show how the variational representation for $f$-divergences leads to novel change of measure inequalities. We also present a multiplicative change of measure inequality for $α$-divergences and a generalized version of Hammersley-Chapman-Robbins inequality. Finally, we present several applications of our change of measure inequalities, including PAC-Bayesian bounds for various classes of losses and non-asymptotic intervals for Monte Carlo estimates.