Perception Prioritized Training of Diffusion Models
This work addresses a key training inefficiency in diffusion models for image generation, offering a simple yet effective method to enhance model performance without major architectural changes.
The paper tackles the problem of inefficient training in diffusion models by prioritizing noise levels that are more informative for learning visual concepts, resulting in significant performance improvements across various datasets, architectures, and sampling strategies.
Diffusion models learn to restore noisy data, which is corrupted with different levels of noise, by optimizing the weighted sum of the corresponding loss terms, i.e., denoising score matching loss. In this paper, we show that restoring data corrupted with certain noise levels offers a proper pretext task for the model to learn rich visual concepts. We propose to prioritize such noise levels over other levels during training, by redesigning the weighting scheme of the objective function. We show that our simple redesign of the weighting scheme significantly improves the performance of diffusion models regardless of the datasets, architectures, and sampling strategies.