Variance reduction of diffusion model's gradients with Taylor approximation-based control variate
This work addresses optimization challenges in diffusion models for machine learning researchers, but it appears incremental as it builds on existing variance reduction techniques.
The authors tackled the high variance in training score-based diffusion models by introducing a control variate derived from a Taylor expansion of the training objective and its gradient, demonstrating its effectiveness empirically in low-dimensional settings and studying its impact on larger problems.
Score-based models, trained with denoising score matching, are remarkably effective in generating high dimensional data. However, the high variance of their training objective hinders optimisation. We attempt to reduce it with a control variate, derived via a $k$-th order Taylor expansion on the training objective and its gradient. We prove an equivalence between the two and demonstrate empirically the effectiveness of our approach on a low dimensional problem setting; and study its effect on larger problems.