Renormalized Mutual Information for Artificial Scientific Discovery
This work addresses the challenge of feature extraction and artificial scientific discovery for researchers in physics and machine learning, though it appears incremental as an extension of mutual information theory.
The authors tackled the problem of estimating dependence between continuous random variables when one is deterministically dependent on the other, deriving a renormalized mutual information method that enables discovery of collective variables in physical systems and aids analysis of information flow in neural networks.
We derive a well-defined renormalized version of mutual information that allows to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation relevant for feature extraction, where the goal is to produce a low-dimensional effective description of a high-dimensional system. Our approach enables the discovery of collective variables in physical systems, thus adding to the toolbox of artificial scientific discovery, while also aiding the analysis of information flow in artificial neural networks.