Emergence of Compositional Representations in Restricted Boltzmann Machines
This work addresses the problem of feature extraction in high-dimensional data for machine learning practitioners, but it is incremental as it builds on known empirical properties of RBMs.
The paper investigates the structural conditions for Restricted Boltzmann Machines to develop compositional representations, such as sparsity and low effective temperature, and validates these findings through replica analysis on random RBMs and training on the MNIST dataset.
Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits dataset MNIST.