Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders
This work provides a theoretical foundation for sampling algorithms in deep learning, addressing a specific bottleneck in auto-encoder-based density estimation, but it is incremental as it builds on existing auto-encoder variants.
The paper tackles the problem of sampling from auto-encoders by developing a mathematical framework for implicit density estimation through local moment matching, showing that auto-encoders with contractive penalties capture local moments and justifying sampling algorithms for Contractive and Denoising Auto-Encoders.
Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder variants. We consider an MCMC where each step samples from a Gaussian whose mean and covariance matrix depend on the previous state, defines through its asymptotic distribution a target density. First, we show that good choices (in the sense of consistency) for these mean and covariance functions are the local expected value and local covariance under that target density. Then we show that an auto-encoder with a contractive penalty captures estimators of these local moments in its reconstruction function and its Jacobian. A contribution of this work is thus a novel alternative to maximum-likelihood density estimation, which we call local moment matching. It also justifies a recently proposed sampling algorithm for the Contractive Auto-Encoder and extends it to the Denoising Auto-Encoder.