Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation
This provides a theoretical foundation for diffusion models, clarifying their optimization for researchers and practitioners.
The authors showed that diffusion model objectives are equivalent to a weighted integral of ELBOs over noise levels, linking them to maximum likelihood under simple data augmentation, and achieved state-of-the-art FID scores on ImageNet.
To achieve the highest perceptual quality, state-of-the-art diffusion models are optimized with objectives that typically look very different from the maximum likelihood and the Evidence Lower Bound (ELBO) objectives. In this work, we reveal that diffusion model objectives are actually closely related to the ELBO. Specifically, we show that all commonly used diffusion model objectives equate to a weighted integral of ELBOs over different noise levels, where the weighting depends on the specific objective used. Under the condition of monotonic weighting, the connection is even closer: the diffusion objective then equals the ELBO, combined with simple data augmentation, namely Gaussian noise perturbation. We show that this condition holds for a number of state-of-the-art diffusion models. In experiments, we explore new monotonic weightings and demonstrate their effectiveness, achieving state-of-the-art FID scores on the high-resolution ImageNet benchmark.