Kernel Embeddings and the Separation of Measure Phenomenon
This provides a theoretical foundation for the kernel trick, potentially improving inference tools in high-dimensional domains, but it is incremental as it builds on existing mathematical frameworks.
The paper proves that kernel covariance embeddings enable perfect separation of distinct probability distributions, showing that testing equality of two probability measures is equivalent to testing singularity between two Gaussian measures in a reproducing kernel Hilbert space, which simplifies information-theoretic analysis compared to non-parametric two-sample testing.
We prove that kernel covariance embeddings lead to information-theoretically perfect separation of distinct probability distributions. In statistical terms, we establish that testing for the equality of two probability measures on a compact and separable metric space is equivalent to testing for the singularity between two centered Gaussian measures on a reproducing kernel Hilbert Space. The corresponding Gaussians are defined via the notion of kernel covariance embedding of a probability measure, and the Hilbert space is that generated by the embedding kernel. Distinguishing singular Gaussians is fundamentally simpler from an information-theoretic perspective than non-parametric two-sample testing, particularly in complex or high-dimensional domains. This is because singular Gaussians are supported on essentially separate and affine subspaces. Our proof leverages the classical Feldman-Hajek dichotomy, and shows that even a small perturbation of a distribution will be maximally magnified through its Gaussian embedding. This ``separation of measure phenomenon'' appears to be a blessing of infinite dimensionality, by means of embedding, with the potential to inform the design of efficient inference tools in considerable generality. The elicitation of this phenomenon also appears to crystallize, in a precise and simple mathematical statement, the outstanding empirical effectiveness of the so-called ``kernel trick".