Is Supervised Learning Really That Different from Unsupervised?
This challenges the fundamental distinction between supervised and unsupervised learning, potentially simplifying model training by reducing reliance on labeled data.
The paper demonstrates that supervised learning can be decomposed into a two-stage unsupervised procedure, where model parameters are selected without access to outputs y, and shows that this approach achieves similar performance to standard supervised methods across various algorithms including linear regression, neural networks, and random forests.
We demonstrate how supervised learning can be decomposed into a two-stage procedure, where (1) all model parameters are selected in an unsupervised manner, and (2) the outputs y are added to the model, without changing the parameter values. This is achieved by a new model selection criterion that - in contrast to cross-validation - can be used also without access to y. For linear ridge regression, we bound the asymptotic out-of-sample risk of our method in terms of the optimal asymptotic risk. We also demonstrate that versions of linear and kernel ridge regression, smoothing splines, k-nearest neighbors, random forests, and neural networks, trained without access to y, perform similarly to their standard y-based counterparts. Hence, our results suggest that the difference between supervised and unsupervised learning is less fundamental than it may appear.