Energy-Inspired Models: Learning with Sampler-Induced Distributions
This addresses the inference mismatch problem in probabilistic modeling for researchers, offering incremental improvements by unifying variational bounds with EIMs.
The paper tackles the intractable sampling and density evaluation in energy-based models by proposing energy-inspired models (EIMs) that use sampler-induced distributions, enabling exact samples and tractable log-likelihood lower bounds, with instantiations outperforming or matching recent methods like Learned Accept/Reject Sampling.
Energy-based models (EBMs) are powerful probabilistic models, but suffer from intractable sampling and density evaluation due to the partition function. As a result, inference in EBMs relies on approximate sampling algorithms, leading to a mismatch between the model and inference. Motivated by this, we consider the sampler-induced distribution as the model of interest and maximize the likelihood of this model. This yields a class of energy-inspired models (EIMs) that incorporate learned energy functions while still providing exact samples and tractable log-likelihood lower bounds. We describe and evaluate three instantiations of such models based on truncated rejection sampling, self-normalized importance sampling, and Hamiltonian importance sampling. These models outperform or perform comparably to the recently proposed Learned Accept/Reject Sampling algorithm and provide new insights on ranking Noise Contrastive Estimation and Contrastive Predictive Coding. Moreover, EIMs allow us to generalize a recent connection between multi-sample variational lower bounds and auxiliary variable variational inference. We show how recent variational bounds can be unified with EIMs as the variational family.