Direct Density-Derivative Estimation and Its Application in KL-Divergence Approximation
This work addresses a bottleneck in statistical data analysis for researchers and practitioners by providing a more accurate and efficient tool for density-derivative estimation, which is incremental as it builds on existing non-parametric methods.
The paper tackles the problem of estimating density derivatives directly without first estimating the density, proposing a method that enables analytic and efficient approximation of multi-dimensional high-order derivatives with objective hyper-parameter selection via cross-validation, and demonstrates its application in improving KL-divergence estimation for tasks like change detection and feature selection.
Estimation of density derivatives is a versatile tool in statistical data analysis. A naive approach is to first estimate the density and then compute its derivative. However, such a two-step approach does not work well because a good density estimator does not necessarily mean a good density-derivative estimator. In this paper, we give a direct method to approximate the density derivative without estimating the density itself. Our proposed estimator allows analytic and computationally efficient approximation of multi-dimensional high-order density derivatives, with the ability that all hyper-parameters can be chosen objectively by cross-validation. We further show that the proposed density-derivative estimator is useful in improving the accuracy of non-parametric KL-divergence estimation via metric learning. The practical superiority of the proposed method is experimentally demonstrated in change detection and feature selection.