Learning Deep Representation Without Parameter Inference for Nonlinear Dimensionality Reduction
This work addresses the challenge of efficient and effective deep learning for researchers and practitioners in machine learning, though it appears incremental as it builds on existing building blocks like product of experts.
The paper tackles the problem of unsupervised deep representation learning by proposing distributed random models as a new building block, which learns better representations than deep belief networks and trains larger networks much faster, with experimental results showing improved performance and reduced training time.
Unsupervised deep learning is one of the most powerful representation learning techniques. Restricted Boltzman machine, sparse coding, regularized auto-encoders, and convolutional neural networks are pioneering building blocks of deep learning. In this paper, we propose a new building block -- distributed random models. The proposed method is a special full implementation of the product of experts: (i) each expert owns multiple hidden units and different experts have different numbers of hidden units; (ii) the model of each expert is a k-center clustering, whose k-centers are only uniformly sampled examples, and whose output (i.e. the hidden units) is a sparse code that only the similarity values from a few nearest neighbors are reserved. The relationship between the pioneering building blocks, several notable research branches and the proposed method is analyzed. Experimental results show that the proposed deep model can learn better representations than deep belief networks and meanwhile can train a much larger network with much less time than deep belief networks.