Value Gradient Sampler: Sampling as Sequential Decision Making
This provides a novel method for sampling and training energy-based models, with applications in areas like anomaly detection, though it appears incremental as it builds on existing reinforcement learning techniques.
The paper tackles the problem of sampling from unnormalized densities by proposing the Value Gradient Sampler (VGS), which interprets sampling as sequential decision-making and uses value-based dynamic programming to optimize drift, achieving competitive results in various benchmarks and enabling effective training of energy-based models for industrial anomaly detection.
We propose the Value Gradient Sampler (VGS), a trainable sampler based on the interpretation of sampling as discrete-time sequential decision-making. VGS generates samples from a given unnormalized density (i.e., energy) by drifting and diffusing randomly initialized particles. In VGS, finding the optimal drift is equivalent to solving an optimal control problem where the cost is the upper bound of the KL divergence between the target density and the samples. We employ value-based dynamic programming to solve this optimal control problem, which gives the gradient of the value function as the optimal drift vector. The connection to sequential decision making allows VGS to leverage extensively studied techniques in reinforcement learning, making VGS a fast, adaptive, and accurate sampler that achieves competitive results in various sampling benchmarks. Furthermore, VGS can replace MCMC in contrastive divergence training of energy-based models. We demonstrate the effectiveness of VGS in training accurate energy-based models in industrial anomaly detection applications.