Discrete Bridges for Mutual Information Estimation
This work addresses a challenge in machine learning and information theory for researchers dealing with discrete data, though it appears incremental as it adapts existing bridge models to a new application.
The authors tackled the problem of estimating mutual information between discrete random variables by framing it as a domain transfer problem, resulting in a Discrete Bridge Mutual Information estimator that performs well in low-dimensional and image-based settings.
Diffusion bridge models in both continuous and discrete state spaces have recently become powerful tools in the field of generative modeling. In this work, we leverage the discrete state space formulation of bridge matching models to address another important problem in machine learning and information theory: the estimation of the mutual information (MI) between discrete random variables. By neatly framing MI estimation as a domain transfer problem, we construct a Discrete Bridge Mutual Information (DBMI) estimator suitable for discrete data, which poses difficulties for conventional MI estimators. We showcase the performance of our estimator on two MI estimation settings: low-dimensional and image-based.