Exponential Concentration of a Density Functional Estimator
This work addresses the need for reliable density functional estimation in statistics and machine learning, offering theoretical guarantees for applications like information theory, but it is incremental as it builds on existing plug-in methods with enhanced concentration results.
The paper tackles the problem of estimating integral functionals of continuous probability densities, such as entropy and mutual information, by analyzing a plug-in estimator. It proves that the estimator converges at a rate of O(n^{-β/(β+d)}) for densities in a β-Hölder smoothness class and shows exponential concentration around its mean, improving upon previous expected error bounds.
We analyze a plug-in estimator for a large class of integral functionals of one or more continuous probability densities. This class includes important families of entropy, divergence, mutual information, and their conditional versions. For densities on the $d$-dimensional unit cube $[0,1]^d$ that lie in a $β$-Hölder smoothness class, we prove our estimator converges at the rate $O \left( n^{-\fracβ{β+ d}} \right)$. Furthermore, we prove the estimator is exponentially concentrated about its mean, whereas most previous related results have proven only expected error bounds on estimators.