Moment Matching Denoising Gibbs Sampling
This addresses a key bottleneck in training and sampling from Energy-Based Models for researchers and practitioners in machine learning, though it appears incremental as it builds on existing DSM methods.
The paper tackles the inconsistency issue in Denoising Score Matching for Energy-Based Models, which leads to learning noisy data distributions, by proposing a moment matching Gibbs sampling framework that enables effective sampling from the underlying clean model, demonstrating scalability to high-dimensional datasets.
Energy-Based Models (EBMs) offer a versatile framework for modeling complex data distributions. However, training and sampling from EBMs continue to pose significant challenges. The widely-used Denoising Score Matching (DSM) method for scalable EBM training suffers from inconsistency issues, causing the energy model to learn a `noisy' data distribution. In this work, we propose an efficient sampling framework: (pseudo)-Gibbs sampling with moment matching, which enables effective sampling from the underlying clean model when given a `noisy' model that has been well-trained via DSM. We explore the benefits of our approach compared to related methods and demonstrate how to scale the method to high-dimensional datasets.