Minimax Optimal Estimation of KL Divergence for Continuous Distributions
This provides a theoretical foundation for KL divergence estimation, which is incremental but important for domains like statistics and machine learning.
The paper tackled the problem of estimating Kullback-Leibler divergence from i.i.d. samples by analyzing the convergence rates of bias and variance for a k-nearest neighbor estimator, showing it is asymptotically rate optimal with a derived minimax lower bound.
Estimating Kullback-Leibler divergence from identical and independently distributed samples is an important problem in various domains. One simple and effective estimator is based on the k nearest neighbor distances between these samples. In this paper, we analyze the convergence rates of the bias and variance of this estimator. Furthermore, we derive a lower bound of the minimax mean square error and show that kNN method is asymptotically rate optimal.