Generalized Exponential Concentration Inequality for Rényi Divergence Estimation
This addresses a fundamental gap in nonparametric statistics for machine learning tasks, though it is incremental as it focuses on a specific class of densities.
The paper tackles the problem of estimating Rényi-α divergence by providing the first finite sample exponential concentration inequality bound for an estimator, with results demonstrated through a numerical experiment.
Estimating divergences in a consistent way is of great importance in many machine learning tasks. Although this is a fundamental problem in nonparametric statistics, to the best of our knowledge there has been no finite sample exponential inequality convergence bound derived for any divergence estimators. The main contribution of our work is to provide such a bound for an estimator of Rényi-$α$ divergence for a smooth Hölder class of densities on the $d$-dimensional unit cube $[0, 1]^d$. We also illustrate our theoretical results with a numerical experiment.