Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks
This work provides theoretical guarantees for diffusion-based generative models, addressing a foundational challenge in machine learning for researchers and practitioners in generative AI.
The paper tackles the problem of approximating ratio-type functionals in conditional generative modeling, such as those in diffusion models, by developing a theoretical framework using SignReLU neural networks, resulting in L^p approximation bounds and excess KL risk bounds for diffusion-based generative models.
Motivated by challenges in conditional generative modeling, where the target conditional density takes the form of a ratio f1 over f2, this paper develops a theoretical framework for approximating such ratio-type functionals. Here, f1 and f2 are kernel-based marginal densities that capture structured interactions, a setting central to diffusion-based generative models. We provide a concise proof for approximating these ratio-type functionals using deep neural networks with the SignReLU activation function, leveraging the activation's piecewise structure. Under standard regularity assumptions, we establish L^p(Omega) approximation bounds and convergence rates. Specializing to Denoising Diffusion Probabilistic Models (DDPMs), we construct a SignReLU-based neural estimator for the reverse process and derive bounds on the excess Kullback-Leibler (KL) risk between the generated and true data distributions. Our analysis decomposes this excess risk into approximation and estimation error components. These results provide generalization guarantees for finite-sample training of diffusion-based generative models.