Optimizing Input of Denoising Score Matching is Biased Towards Higher Score Norm
This work identifies a critical bias affecting various applications like auto-regressive generation, image compression, and text-to-3D generation, highlighting a fundamental issue in diffusion model optimization.
The paper demonstrates that optimizing the conditional input in denoising score matching for diffusion models introduces a bias, breaking the equivalence with exact score matching and leading to higher score norms, with similar effects observed when optimizing data distributions using pre-trained models.
Many recent works utilize denoising score matching to optimize the conditional input of diffusion models. In this workshop paper, we demonstrate that such optimization breaks the equivalence between denoising score matching and exact score matching. Furthermore, we show that this bias leads to higher score norm. Additionally, we observe a similar bias when optimizing the data distribution using a pre-trained diffusion model. Finally, we discuss the wide range of works across different domains that are affected by this bias, including MAR for auto-regressive generation, PerCo for image compression, and DreamFusion for text to 3D generation.