Nonparanormal Information Estimation
This addresses the lack of robust mutual information estimators for realistic data, which is crucial for applications in statistics and machine learning, though it is an incremental improvement over existing semiparametric approaches.
The paper tackles the problem of estimating mutual information from multivariate data, where existing estimators are either brittle under non-Gaussian assumptions or fail to scale with realistic sample sizes. The authors propose estimators based on a nonparanormal model, showing they achieve a practical balance between robustness and dimensionality scaling through theoretical bounds and experiments.
We study the problem of using i.i.d. samples from an unknown multivariate probability distribution $p$ to estimate the mutual information of $p$. This problem has recently received attention in two settings: (1) where $p$ is assumed to be Gaussian and (2) where $p$ is assumed only to lie in a large nonparametric smoothness class. Estimators proposed for the Gaussian case converge in high dimensions when the Gaussian assumption holds, but are brittle, failing dramatically when $p$ is not Gaussian. Estimators proposed for the nonparametric case fail to converge with realistic sample sizes except in very low dimensions. As a result, there is a lack of robust mutual information estimators for many realistic data. To address this, we propose estimators for mutual information when $p$ is assumed to be a nonparanormal (a.k.a., Gaussian copula) model, a semiparametric compromise between Gaussian and nonparametric extremes. Using theoretical bounds and experiments, we show these estimators strike a practical balance between robustness and scaling with dimensionality.