On Distance and Kernel Measures of Conditional Independence
This work addresses a theoretical problem in statistical inference for researchers in causal discovery and machine learning, but it is incremental as it builds on prior work on joint independence.
The paper investigates the relationship between distance-based and kernel-based measures of conditional independence, showing equivalence for certain pairs but not for popular kernel measures like those based on Hilbert-Schmidt norms, except in limiting cases.
Measuring conditional independence is one of the important tasks in statistical inference and is fundamental in causal discovery, feature selection, dimensionality reduction, Bayesian network learning, and others. In this work, we explore the connection between conditional independence measures induced by distances on a metric space and reproducing kernels associated with a reproducing kernel Hilbert space (RKHS). For certain distance and kernel pairs, we show the distance-based conditional independence measures to be equivalent to that of kernel-based measures. On the other hand, we also show that some popular---in machine learning---kernel conditional independence measures based on the Hilbert-Schmidt norm of a certain cross-conditional covariance operator, do not have a simple distance representation, except in some limiting cases. This paper, therefore, shows the distance and kernel measures of conditional independence to be not quite equivalent unlike in the case of joint independence as shown by Sejdinovic et al. (2013).