An Analysis of the Connections Between Layers of Deep Neural Networks
This work addresses efficiency in unsupervised deep neural networks, but it is incremental as it builds on existing connection methods.
The paper tackled the problem of selecting connections between layers in unsupervised deep neural networks, where back-propagation is unavailable, and found that certain techniques improved accuracy by up to 3% on CIFAR and SVHN datasets.
We present an analysis of different techniques for selecting the connection be- tween layers of deep neural networks. Traditional deep neural networks use ran- dom connection tables between layers to keep the number of connections small and tune to different image features. This kind of connection performs adequately in supervised deep networks because their values are refined during the training. On the other hand, in unsupervised learning, one cannot rely on back-propagation techniques to learn the connections between layers. In this work, we tested four different techniques for connecting the first layer of the network to the second layer on the CIFAR and SVHN datasets and showed that the accuracy can be im- proved up to 3% depending on the technique used. We also showed that learning the connections based on the co-occurrences of the features does not confer an advantage over a random connection table in small networks. This work is helpful to improve the efficiency of connections between the layers of unsupervised deep neural networks.