Deep Dimension Reduction for Supervised Representation Learning
This work addresses the challenge of learning effective nonparametric representations for high-dimensional data in machine learning, offering incremental improvements over existing methods.
The authors tackled the problem of supervised representation learning by proposing a deep dimension reduction approach to create representations that are sufficient, low-dimensional, and disentangled, and demonstrated through experiments that their method outperforms existing dimension reduction techniques and standard deep learning models in classification and regression tasks.
The goal of supervised representation learning is to construct effective data representations for prediction. Among all the characteristics of an ideal nonparametric representation of high-dimensional complex data, sufficiency, low dimensionality and disentanglement are some of the most essential ones. We propose a deep dimension reduction approach to learning representations with these characteristics. The proposed approach is a nonparametric generalization of the sufficient dimension reduction method. We formulate the ideal representation learning task as that of finding a nonparametric representation that minimizes an objective function characterizing conditional independence and promoting disentanglement at the population level. We then estimate the target representation at the sample level nonparametrically using deep neural networks. We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero. Our extensive numerical experiments using simulated and real benchmark data demonstrate that the proposed methods have better performance than several existing dimension reduction methods and the standard deep learning models in the context of classification and regression.