Estimating Rényi's $α$-Cross-Entropies in a Matrix-Based Way
This provides a method for assessing distributional differences in high-dimensional data, which is incremental as it builds on existing Rényi divergence theory with a new computational approach.
The paper tackled the problem of estimating Rényi's α-cross-entropies without requiring prior probability distribution estimation by proposing three matrix-based measures in reproducing-kernel Hilbert spaces, achieving unbiased, non-parametric, and minimax-optimal estimators with convergence rates independent of sample dimensionality.
Conventional information-theoretic quantities assume access to probability distributions. Estimating such distributions is not trivial. Here, we consider function-based formulations of cross entropy that sidesteps this a priori estimation requirement. We propose three measures of Rényi's $α$-cross-entropies in the setting of reproducing-kernel Hilbert spaces. Each measure has its appeals. We prove that we can estimate these measures in an unbiased, non-parametric, and minimax-optimal way. We do this via sample-constructed Gram matrices. This yields matrix-based estimators of Rényi's $α$-cross-entropies. These estimators satisfy all of the axioms that Rényi established for divergences. Our cross-entropies can thus be used for assessing distributional differences. They are also appropriate for handling high-dimensional distributions, since the convergence rate of our estimator is independent of the sample dimensionality. Python code for implementing these measures can be found at https://github.com/isledge/MBRCE