Marginalizable Density Models
This addresses a key limitation for researchers and practitioners in machine learning who need tractable probabilistic inference, though it is an incremental improvement over existing neural density models.
The paper tackles the problem of deep density models lacking closed-form marginals and conditionals by introducing the Marginalizable Density Model Approximator (MDMA), which provides exact expressions for these quantities and outperforms state-of-the-art models in tasks like mutual information estimation and causal inference.
Probability density models based on deep networks have achieved remarkable success in modeling complex high-dimensional datasets. However, unlike kernel density estimators, modern neural models do not yield marginals or conditionals in closed form, as these quantities require the evaluation of seldom tractable integrals. In this work, we present the Marginalizable Density Model Approximator (MDMA), a novel deep network architecture which provides closed form expressions for the probabilities, marginals and conditionals of any subset of the variables. The MDMA learns deep scalar representations for each individual variable and combines them via learned hierarchical tensor decompositions into a tractable yet expressive CDF, from which marginals and conditional densities are easily obtained. We illustrate the advantage of exact marginalizability in several tasks that are out of reach of previous deep network-based density estimation models, such as estimating mutual information between arbitrary subsets of variables, inferring causality by testing for conditional independence, and inference with missing data without the need for data imputation, outperforming state-of-the-art models on these tasks. The model also allows for parallelized sampling with only a logarithmic dependence of the time complexity on the number of variables.