DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
This addresses the need for efficient and controllable image generation in computer vision, though it is incremental as it combines existing VAE and diffusion components.
The paper tackles the problem of diffusion models lacking low-dimensional latent spaces and being slow at generation, while VAEs have poor sample quality, by proposing DiffuseVAE which integrates VAE within a diffusion framework to achieve efficient, controllable, and high-fidelity generation. The result shows improved speed-quality tradeoffs (e.g., FID of 16.47 vs 34.36 on CelebA-HQ-128 with T=10 steps) and competitive synthesis quality on benchmarks like CIFAR-10 and CelebA-64.
Diffusion probabilistic models have been shown to generate state-of-the-art results on several competitive image synthesis benchmarks but lack a low-dimensional, interpretable latent space, and are slow at generation. On the other hand, standard Variational Autoencoders (VAEs) typically have access to a low-dimensional latent space but exhibit poor sample quality. We present DiffuseVAE, a novel generative framework that integrates VAE within a diffusion model framework, and leverage this to design novel conditional parameterizations for diffusion models. We show that the resulting model equips diffusion models with a low-dimensional VAE inferred latent code which can be used for downstream tasks like controllable synthesis. The proposed method also improves upon the speed vs quality tradeoff exhibited in standard unconditional DDPM/DDIM models (for instance, FID of 16.47 vs 34.36 using a standard DDIM on the CelebA-HQ-128 benchmark using T=10 reverse process steps) without having explicitly trained for such an objective. Furthermore, the proposed model exhibits synthesis quality comparable to state-of-the-art models on standard image synthesis benchmarks like CIFAR-10 and CelebA-64 while outperforming most existing VAE-based methods. Lastly, we show that the proposed method exhibits inherent generalization to different types of noise in the conditioning signal. For reproducibility, our source code is publicly available at https://github.com/kpandey008/DiffuseVAE.