Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions
This work challenges the need for complex models in high-dimensional analysis, suggesting that simpler linear approaches may suffice under certain conditions, which is incremental as it builds on existing asymptotic analyses.
The paper shows that in high-dimensional regimes with independent covariates, kernel methods and multi-layer perceptrons perform no better than linear models, and linear models are optimal even when data is generated by a nonlinear kernel model.
Empirical observation of high dimensional phenomena, such as the double descent behaviour, has attracted a lot of interest in understanding classical techniques such as kernel methods, and their implications to explain generalization properties of neural networks. Many recent works analyze such models in a certain high-dimensional regime where the covariates are independent and the number of samples and the number of covariates grow at a fixed ratio (i.e. proportional asymptotics). In this work we show that for a large class of kernels, including the neural tangent kernel of fully connected networks, kernel methods can only perform as well as linear models in this regime. More surprisingly, when the data is generated by a kernel model where the relationship between input and the response could be very nonlinear, we show that linear models are in fact optimal, i.e. linear models achieve the minimum risk among all models, linear or nonlinear. These results suggest that more complex models for the data other than independent features are needed for high-dimensional analysis.