MADE: Masked Autoencoder for Distribution Estimation
This provides a faster and scalable method for distribution estimation, which is useful for generative modeling tasks, though it is incremental as it builds on existing autoencoder and autoregressive frameworks.
The paper tackles the problem of estimating distributions from data by introducing a masked autoencoder that enforces autoregressive constraints, allowing it to output conditional probabilities and compute joint probabilities efficiently. Experiments show it is competitive with state-of-the-art tractable estimators, with significantly faster test-time performance and better scaling.
There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.