Why Unsupervised Deep Networks Generalize
This work offers a theoretical explanation for the generalization capabilities of deep networks, which is a fundamental problem in machine learning. It also provides a practical method for faster autoencoder training.
This paper proposes that deep networks generalize because they discard high momentum modes, similar to the renormalization group. For an RBM, the authors provide quantitative evidence for this hypothesis. They also present an algorithm to directly determine autoencoder parameters from data, achieving near deep learning performance and reducing training times by 4-100x.
Promising resolutions of the generalization puzzle observe that the actual number of parameters in a deep network is much smaller than naive estimates suggest. The renormalization group is a compelling example of a problem which has very few parameters, despite the fact that naive estimates suggest otherwise. Our central hypothesis is that the mechanisms behind the renormalization group are also at work in deep learning, and that this leads to a resolution of the generalization puzzle. We show detailed quantitative evidence that proves the hypothesis for an RBM, by showing that the trained RBM is discarding high momentum modes. Specializing attention mainly to autoencoders, we give an algorithm to determine the network's parameters directly from the learning data set. The resulting autoencoder almost performs as well as one trained by deep learning, and it provides an excellent initial condition for training, reducing training times by a factor between 4 and 100 for the experiments we considered. Further, we are able to suggest a simple criterion to decide if a given problem can or can not be solved using a deep network.