Stochastic Neural Networks with Monotonic Activation Functions
This work provides a method for training stochastic neural networks with monotonic activations, which is incremental as it builds on existing RBM and Harmonium frameworks.
The paper tackled the problem of creating stochastic neural units from smooth monotonic activation functions using a Laplace approximation with Gaussian noise, and demonstrated that the resulting exponential family Restricted Boltzmann Machines (Exp-RBM) can learn useful representations through contrastive divergence.
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences. This family, that we call exponential family RBM (Exp-RBM), is a subset of the exponential family Harmoniums that expresses family members through a choice of smooth monotonic non-linearity for each neuron. Using contrastive divergence along with our Gaussian approximation, we show that Exp-RBM can learn useful representations using novel stochastic units.