PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
This work provides incremental improvements to generative modeling for researchers and practitioners in machine learning.
The paper tackles improving PixelCNN generative models by introducing modifications like discretized logistic mixture likelihood and downsampling, resulting in state-of-the-art log likelihood results on CIFAR-10.
PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at https://github.com/openai/pixel-cnn. Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R/G/B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.