A Probabilistic Framework for Nonlinearities in Stochastic Neural Networks
This work addresses the challenge of optimizing nonlinearities in stochastic neural networks, offering a unified approach that could benefit researchers in machine learning, though it appears incremental as it builds on existing stochastic models.
The authors tackled the problem of learning nonlinearities in stochastic neural networks by introducing a probabilistic framework based on doubly truncated Gaussian distributions, which improved performance in models like RBM, temporal RBM, and TGGM.
We present a probabilistic framework for nonlinearities, based on doubly truncated Gaussian distributions. By setting the truncation points appropriately, we are able to generate various types of nonlinearities within a unified framework, including sigmoid, tanh and ReLU, the most commonly used nonlinearities in neural networks. The framework readily integrates into existing stochastic neural networks (with hidden units characterized as random variables), allowing one for the first time to learn the nonlinearities alongside model weights in these networks. Extensive experiments demonstrate the performance improvements brought about by the proposed framework when integrated with the restricted Boltzmann machine (RBM), temporal RBM and the truncated Gaussian graphical model (TGGM).