Kullback-Leibler and Renyi divergences in reproducing kernel Hilbert space and Gaussian process settings
This provides a theoretical foundation for divergence estimation in machine learning, particularly for kernel methods and Gaussian processes, but is incremental as it builds on existing RKHS methodology.
The paper tackles the problem of estimating infinite-dimensional divergences (Kullback-Leibler and Rényi) in reproducing kernel Hilbert space and Gaussian process settings, showing that these divergences can be consistently and efficiently estimated from finite-dimensional versions with dimension-independent sample complexities.
In this work, we present formulations for regularized Kullback-Leibler and Rényi divergences via the Alpha Log-Determinant (Log-Det) divergences between positive Hilbert-Schmidt operators on Hilbert spaces in two different settings, namely (i) covariance operators and Gaussian measures defined on reproducing kernel Hilbert spaces (RKHS); and (ii) Gaussian processes with squared integrable sample paths. For characteristic kernels, the first setting leads to divergences between arbitrary Borel probability measures on a complete, separable metric space. We show that the Alpha Log-Det divergences are continuous in the Hilbert-Schmidt norm, which enables us to apply laws of large numbers for Hilbert space-valued random variables. As a consequence of this, we show that, in both settings, the infinite-dimensional divergences can be consistently and efficiently estimated from their finite-dimensional versions, using finite-dimensional Gram matrices/Gaussian measures and finite sample data, with {\it dimension-independent} sample complexities in all cases. RKHS methodology plays a central role in the theoretical analysis in both settings. The mathematical formulation is illustrated by numerical experiments.