On Compression of Unsupervised Neural Nets by Pruning Weak Connections
This addresses the issue of model size and efficiency for practitioners using unsupervised neural networks, though it is incremental as it builds on existing pruning techniques.
The paper tackles the problem of compressing unsupervised neural networks like RBMs and DBNs by pruning weak connections, resulting in dramatically reduced parameters with virtually no loss in generative or discriminative performance.
Unsupervised neural nets such as Restricted Boltzmann Machines(RBMs) and Deep Belif Networks(DBNs), are powerful in automatic feature extraction,unsupervised weight initialization and density estimation. In this paper,we demonstrate that the parameters of these neural nets can be dramatically reduced without affecting their performance. We describe a method to reduce the parameters required by RBM which is the basic building block for deep architectures. Further we propose an unsupervised sparse deep architectures selection algorithm to form sparse deep neural networks.Experimental results show that there is virtually no loss in either generative or discriminative performance.