Iterative Neural Autoregressive Distribution Estimator (NADE-k)
This work addresses density estimation for unsupervised deep learning, offering incremental improvements over existing methods like NADE.
The paper tackles the problem of improving density estimation by extending the neural autoregressive density estimator (NADE) to multiple inference steps, arguing it's easier to learn incremental improvements. The result is NADE-k, which is competitive with state-of-the-art methods on two datasets.
Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in $k$ steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-predictive training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.