On the Effectiveness of Hybrid Mutual Information Estimation
This work addresses a fundamental challenge in information theory and machine learning for researchers and practitioners, but it is incremental as it builds on existing variational bounds and methods.
The authors tackled the problem of estimating mutual information from samples by proposing a hybrid method that combines discriminative and generative approaches, along with a new generative technique called Predictive Quantization, resulting in tighter bounds and improved estimates on challenging tasks like high-dimensional Gaussian distributions and stochastic processes.
Estimating the mutual information from samples from a joint distribution is a challenging problem in both science and engineering. In this work, we realize a variational bound that generalizes both discriminative and generative approaches. Using this bound, we propose a hybrid method to mitigate their respective shortcomings. Further, we propose Predictive Quantization (PQ): a simple generative method that can be easily combined with discriminative estimators for minimal computational overhead. Our propositions yield a tighter bound on the information thanks to the reduced variance of the estimator. We test our methods on a challenging task of correlated high-dimensional Gaussian distributions and a stochastic process involving a system of free particles subjected to a fixed energy landscape. Empirical results show that hybrid methods consistently improved mutual information estimates when compared to the corresponding discriminative counterpart.