Diffeomorphic Information Neural Estimation
This addresses a fundamental problem in statistics and machine learning for researchers and practitioners needing reliable dependency measures, though it appears incremental as it builds on existing estimation challenges with a new method.
The paper tackled the challenge of estimating conditional mutual information (CMI) for continuous random variables, which is difficult due to intractable formulations, by introducing DINE, a novel estimator that uses diffeomorphic maps to simplify distributions and enable efficient evaluation; empirical results show it consistently outperforms state-of-the-art methods in tasks like MI and CMI estimation and conditional independence testing, adapting well to complex, high-dimensional relationships.
Mutual Information (MI) and Conditional Mutual Information (CMI) are multi-purpose tools from information theory that are able to naturally measure the statistical dependencies between random variables, thus they are usually of central interest in several statistical and machine learning tasks, such as conditional independence testing and representation learning. However, estimating CMI, or even MI, is infamously challenging due the intractable formulation. In this study, we introduce DINE (Diffeomorphic Information Neural Estimator)-a novel approach for estimating CMI of continuous random variables, inspired by the invariance of CMI over diffeomorphic maps. We show that the variables of interest can be replaced with appropriate surrogates that follow simpler distributions, allowing the CMI to be efficiently evaluated via analytical solutions. Additionally, we demonstrate the quality of the proposed estimator in comparison with state-of-the-arts in three important tasks, including estimating MI, CMI, as well as its application in conditional independence testing. The empirical evaluations show that DINE consistently outperforms competitors in all tasks and is able to adapt very well to complex and high-dimensional relationships.