Mutual Information Multinomial Estimation
This work addresses a fundamental problem in data science and machine learning, but appears incremental as it builds on existing estimation methods.
The paper tackles the challenging problem of estimating mutual information by proposing a new estimator that uses a preliminary estimate of the data distribution as a bridge to compute the difference between joint and marginal distributions, showing advantages in experiments on synthetic and real-world tasks.
Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. This work proposes a new estimator for mutual information. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. This preliminary estimate serves as a bridge between the joint and the marginal distribution, and by comparing with this bridge distribution we can easily obtain the true difference between the joint distributions and the marginal distributions. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method.