Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow
This addresses the problem of accurate likelihood estimation for discrete data in flow models, which is incremental but improves performance in generative modeling.
The paper tackles the dequantization gap in flow models for discrete data by introducing subset flows, which allow exact likelihood computation, and identifies autoregressive models as single-layer flows, achieving state-of-the-art results on CIFAR-10.
Flow models have recently made great progress at modeling ordinal discrete data such as images and audio. Due to the continuous nature of flow models, dequantization is typically applied when using them for such discrete data, resulting in lower bound estimates of the likelihood. In this paper, we introduce subset flows, a class of flows that can tractably transform finite volumes and thus allow exact computation of likelihoods for discrete data. Based on subset flows, we identify ordinal discrete autoregressive models, including WaveNets, PixelCNNs and Transformers, as single-layer flows. We use the flow formulation to compare models trained and evaluated with either the exact likelihood or its dequantization lower bound. Finally, we study multilayer flows composed of PixelCNNs and non-autoregressive coupling layers and demonstrate state-of-the-art results on CIFAR-10 for flow models trained with dequantization.