Generative Modeling with Denoising Auto-Encoders and Langevin Sampling
This work addresses theoretical gaps in generative modeling for researchers, offering incremental theoretical justification for empirical methods.
The paper tackles the problem of analyzing convergence in generative modeling using denoising auto-encoders or denoising score matching with Langevin sampling, providing finite-sample bounds in Wasserstein distance and applying results to justify an existing homotopy method.
We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes. We overcome the challenge of relying only on $L^2$ bounds on the score estimation error and provide finite-sample bounds in the Wasserstein distance between the law of the population distribution and the law of this sampling scheme. We then apply our results to the homotopy method of arXiv:1907.05600 and provide theoretical justification for its empirical success.