A Neural Difference-of-Entropies Estimator for Mutual Information
This work addresses a key problem in statistical learning for researchers and practitioners, but it is incremental as it builds on existing normalizing flow methods.
The paper tackles the challenge of estimating Mutual Information in high dimensions by proposing a novel estimator based on normalizing flows, achieving improved bias-variance trade-offs on standard benchmarks.
Estimating Mutual Information (MI), a key measure of dependence of random quantities without specific modelling assumptions, is a challenging problem in high dimensions. We propose a novel mutual information estimator based on parametrizing conditional densities using normalizing flows, a deep generative model that has gained popularity in recent years. This estimator leverages a block autoregressive structure to achieve improved bias-variance trade-offs on standard benchmark tasks.