MMG: Mutual Information Estimation via the MMSE Gap in Diffusion
This provides a scalable and improved MI estimation method for researchers and practitioners in machine learning and information theory, though it is incremental as it builds on existing diffusion model frameworks.
The paper tackles the challenge of estimating mutual information (MI) for complex systems by proposing a method based on denoising diffusion models, showing that MI corresponds to half the MMSE gap integrated over SNRs, and it outperforms traditional and score-based diffusion MI estimators while maintaining strong performance at high MI.
Mutual information (MI) is one of the most general ways to measure relationships between random variables, but estimating this quantity for complex systems is challenging. Denoising diffusion models have recently set a new bar for density estimation, so it is natural to consider whether these methods could also be used to improve MI estimation. Using the recently introduced information-theoretic formulation of denoising diffusion models, we show the diffusion models can be used in a straightforward way to estimate MI. In particular, the MI corresponds to half the gap in the Minimum Mean Square Error (MMSE) between conditional and unconditional diffusion, integrated over all Signal-to-Noise-Ratios (SNRs) in the noising process. Our approach not only passes self-consistency tests but also outperforms traditional and score-based diffusion MI estimators. Furthermore, our method leverages adaptive importance sampling to achieve scalable MI estimation, while maintaining strong performance even when the MI is high.