Improving Variational Inference with Inverse Autoregressive Flow
This work addresses the bottleneck of flexible variational inference for high-dimensional data in machine learning, offering a novel method that is competitive with existing models while improving efficiency.
The authors tackled the problem of scaling variational inference to high-dimensional latent spaces by introducing inverse autoregressive flow (IAF), a new normalizing flow based on autoregressive neural networks. The result showed that IAF significantly improves upon diagonal Gaussian approximate posteriors and enables a variational autoencoder to achieve competitive log-likelihood on natural images with faster synthesis.
The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis.