Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
This work addresses a fundamental bottleneck in probabilistic modeling for researchers and practitioners, offering a method to handle intractable models without structural assumptions, though it is incremental in building on existing variational and score-based techniques.
The paper tackles the challenge of learning and evaluating energy-based latent variable models (EBLVMs) with intractable posteriors and partition functions by introducing variational estimates of the score function and its gradient (VaES and VaGES). It shows that these estimates can be applied to kernelized Stein discrepancy and score matching methods for learning EBLVMs, with theoretical bounds on bias.
The learning and evaluation of energy-based latent variable models (EBLVMs) without any structural assumptions are highly challenging, because the true posteriors and the partition functions in such models are generally intractable. This paper presents variational estimates of the score function and its gradient with respect to the model parameters in a general EBLVM, referred to as VaES and VaGES respectively. The variational posterior is trained to minimize a certain divergence to the true model posterior and the bias in both estimates can be bounded by the divergence theoretically. With a minimal model assumption, VaES and VaGES can be applied to the kernelized Stein discrepancy (KSD) and score matching (SM)-based methods to learn EBLVMs. Besides, VaES can also be used to estimate the exact Fisher divergence between the data and general EBLVMs.