Class-frequency Guided Noise Schedule for Diffusion Models
For practitioners using diffusion models on imbalanced datasets, this work provides a simple yet effective noise schedule adjustment to improve generation quality for rare classes.
This paper identifies that low-frequency classes suffer from larger low-density regions in diffusion models, leading to poorer generation quality. The proposed Class-frequency Guided (CFRG) noise schedule assigns larger-scale noises to low-frequency classes, achieving substantial improvements over baselines on imbalanced datasets like CIFAR-100-LT and ImageNet-LT.
In this paper, we are the first to examine the correlations between class frequency and the multi-scale noise schedule within diffusion models. For score-based generative models, low-density regions often lead to inaccurately estimated scores, thereby compromising the generation quality. Although the multi-scale noise schedule can alleviate this issue during the diffusion process, low-frequency classes still face the challenge of large low-density regions, resulting in more inaccurate estimated scores than high-frequency classes. Furthermore, high-frequency classes tend to dominate the score space, causing a convergence of most data points towards generating samples from these classes. Consequently, samples generated within low-frequency classes exhibit suboptimal quality and limited diversity. To address this challenge, we propose the \textit{Class-frequency Guided (CFRG)} noise schedule, leveraging the insight that low-frequency classes should be endowed with larger-scale noises. To illustrate the effectiveness of our method, we conduct experiments on various tasks, including image generation, image classification, and text-to-image generation, using imbalanced datasets, \textit{i.e.}, CIFAR-100-LT, and ImageNet-LT. By employing the CFRG noise schedule, we achieve substantial improvements over baselines, manifesting the crucial role of frequency statistics in noise schedule design.