Le Cam meets LeCun: Deficiency and Generic Feature Learning
This work addresses the foundational challenge of unsupervised feature learning for machine learning practitioners, offering theoretical insights into deep learning methods.
The paper tackles the problem of defining and learning generic features from unlabeled data, characterizing when such learning is possible and proposing methods related to autoencoders and deep belief networks.
"Deep Learning" methods attempt to learn generic features in an unsupervised fashion from a large unlabelled data set. These generic features should perform as well as the best hand crafted features for any learning problem that makes use of this data. We provide a definition of generic features, characterize when it is possible to learn them and provide methods closely related to the autoencoder and deep belief network of deep learning. In order to do so we use the notion of deficiency and illustrate its value in studying certain general learning problems.