Gaussian-Bernoulli RBMs Without Tears
This addresses a specific bottleneck in training GRBMs for researchers in generative modeling, though it appears incremental as it builds on existing RBM methods.
The paper tackles the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs) by introducing a novel Gibbs-Langevin sampling algorithm and a modified contrastive divergence algorithm, enabling robust training with large learning rates and generating good samples on datasets like MNIST, FashionMNIST, and CelebA.
We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling algorithm that outperforms existing methods like Gibbs sampling. We propose a modified contrastive divergence (CD) algorithm so that one can generate images with GRBMs starting from noise. This enables direct comparison of GRBMs with deep generative models, improving evaluation protocols in the RBM literature. Moreover, we show that modified CD and gradient clipping are enough to robustly train GRBMs with large learning rates, thus removing the necessity of various tricks in the literature. Experiments on Gaussian Mixtures, MNIST, FashionMNIST, and CelebA show GRBMs can generate good samples, despite their single-hidden-layer architecture. Our code is released at: \url{https://github.com/lrjconan/GRBM}.