Information Theory in Density Destructors
This work offers a new approach for more accurate estimation of information theoretic quantities, which is important for researchers and practitioners working with complex data distributions.
This paper explores density destructors, differentiable and invertible transforms that map complex multivariate probability density functions (PDFs) to maximum entropy PDFs. It demonstrates their connection to information theory and shows they can provide more accurate estimates of information theoretic quantities like total correlation and mutual information compared to existing methods.
Density destructors are differentiable and invertible transforms that map multivariate PDFs of arbitrary structure (low entropy) into non-structured PDFs (maximum entropy). Multivariate Gaussianization and multivariate equalization are specific examples of this family, which break down the complexity of the original PDF through a set of elementary transforms that progressively remove the structure of the data. We demonstrate how this property of density destructive flows is connected to classical information theory, and how density destructors can be used to get more accurate estimates of information theoretic quantities. Experiments with total correlation and mutual information inmultivariate sets illustrate the ability of density destructors compared to competing methods. These results suggest that information theoretic measures may be an alternative optimization criteria when learning density destructive flows.