Recovery of Joint Probability Distribution from one-way marginals: Low rank Tensors and Random Projections
This work addresses a fundamental challenge in machine learning for probabilistic modeling, offering a more efficient approach by reducing the need for higher-order marginals, though it builds on existing low-rank and projection techniques.
The paper tackles the problem of joint probability mass function estimation, which suffers from exponential parameter scaling, by proposing a method to recover the joint density from only one-way marginals using low-rank tensor decomposition and random projections. The result is a novel algorithm tested on synthetic and real-world datasets, achieving effective estimation and classification through MAP inference.
Joint probability mass function (PMF) estimation is a fundamental machine learning problem. The number of free parameters scales exponentially with respect to the number of random variables. Hence, most work on nonparametric PMF estimation is based on some structural assumptions such as clique factorization adopted by probabilistic graphical models, imposition of low rank on the joint probability tensor and reconstruction from 3-way or 2-way marginals, etc. In the present work, we link random projections of data to the problem of PMF estimation using ideas from tomography. We integrate this idea with the idea of low-rank tensor decomposition to show that we can estimate the joint density from just one-way marginals in a transformed space. We provide a novel algorithm for recovering factors of the tensor from one-way marginals, test it across a variety of synthetic and real-world datasets, and also perform MAP inference on the estimated model for classification.