Incorporating Inductive Biases to Energy-based Generative Models
This work addresses the problem of enhancing generative modeling for researchers by providing an incremental method to integrate prior knowledge into EBMs.
The paper tackles the challenge of incorporating inductive biases into energy-based generative models by introducing a hybrid approach that combines an EBM with an exponential family model, resulting in improved data fitting and generation when suitable statistics are used.
With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling. Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics. Like an exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approximately maximizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model's ability to match statistics. Furthermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model.