Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
This work addresses the theoretical understanding of optimization dynamics in score-based generative models, which is important for practitioners training these models, though the results are incremental as they extend existing optimization theory to a specific objective.
The paper provides non-asymptotic convergence guarantees for stochastic gradient descent (SGD) in training score-based generative models, including a convergence rate for general score parameterizations and error bounds for overparameterized two-layer ReLU networks via a Neural Tangent Kernel analysis. The results quantify the effect of the reweighting factor on score approximation error, offering theoretical guidance for practical weighting choices.
Score-based Generative Models (SGMs) have achieved impressive performance in data generation across a wide range of applications. While the statistical properties of their sampling procedures are increasingly well understood, the optimization dynamics underlying their training remain less explored. SGMs are typically trained by minimizing a weighted denoising scorematching objective, yet optimization guarantees with stochastic gradients remain limited. In this work, we study Stochastic Gradient Descent (SGD) for SGMs, contributing results in two complementary regimes. First, for general score parameterizations, we establish a non-convex convergence rate for SGD on the weighted denoising score-matching objective, with explicit dependence on the schedule-dependent weighting factors. Second, for overparameterized two-layer ReLU networks, we develop a Neural Tangent Kernel analysis tailored to diffusion training with stochastic gradients, yielding score-approximation error bounds along the SGD trajectory. Finally, our analysis quantifies the role of the reweighting factor in the score approximation error, providing theoretical guidance for weighting choices used in practice.