Analysis of KNN Information Estimators for Smooth Distributions
This work addresses theoretical gaps in information estimation for practitioners using KSG in broader real-world scenarios, though it is incremental as it extends existing analyses.
The paper analyzes the convergence rate of the KSG mutual information estimator for smooth distributions with both bounded and unbounded support, extending previous analyses limited to densities bounded away from zero, and also provides convergence analysis for the KL entropy estimator.
KSG mutual information estimator, which is based on the distances of each sample to its k-th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence rate of this estimator for random variables whose densities are bounded away from zero in its support. In practice, however, KSG estimator also performs well for a much broader class of distributions, including not only those with bounded support and densities bounded away from zero, but also those with bounded support but densities approaching zero, and those with unbounded support. In this paper, we analyze the convergence rate of the error of KSG estimator for smooth distributions, whose support of density can be both bounded and unbounded. As KSG mutual information estimator can be viewed as an adaptive recombination of KL entropy estimators, in our analysis, we also provide convergence analysis of KL entropy estimator for a broad class of distributions.