Annealing Gaussian into ReLU: a New Sampling Strategy for Leaky-ReLU RBM
This work addresses a specific challenge in training energy-based generative models for machine learning researchers, offering an incremental improvement in sampling efficiency.
The authors tackled the problem of sampling from leaky ReLU Restricted Boltzmann Machines by proposing a method that anneals leakiness instead of temperature, resulting in more efficient and accurate likelihood estimation than annealed importance sampling and faster mixing than contrastive divergence.
Restricted Boltzmann Machine (RBM) is a bipartite graphical model that is used as the building block in energy-based deep generative models. Due to numerical stability and quantifiability of the likelihood, RBM is commonly used with Bernoulli units. Here, we consider an alternative member of exponential family RBM with leaky rectified linear units -- called leaky RBM. We first study the joint and marginal distributions of leaky RBM under different leakiness, which provides us important insights by connecting the leaky RBM model and truncated Gaussian distributions. The connection leads us to a simple yet efficient method for sampling from this model, where the basic idea is to anneal the leakiness rather than the energy; -- i.e., start from a fully Gaussian/Linear unit and gradually decrease the leakiness over iterations. This serves as an alternative to the annealing of the temperature parameter and enables numerical estimation of the likelihood that are more efficient and more accurate than the commonly used annealed importance sampling (AIS). We further demonstrate that the proposed sampling algorithm enjoys faster mixing property than contrastive divergence algorithm, which benefits the training without any additional computational cost.