Asymptotic Optimism of Random-Design Linear and Kernel Regression Models
This work provides theoretical grounding for a new complexity measure in regression, but it is incremental as it builds on existing optimism concepts.
The authors derived closed-form asymptotic optimism for linear and kernel ridge regression under random designs, and used it as a predictive complexity measure to show that neural networks with ReLUs behave differently from kernel models, with empirical validation using resampling techniques on real data.
We derived the closed-form asymptotic optimism of linear regression models under random designs, and generalizes it to kernel ridge regression. Using scaled asymptotic optimism as a generic predictive model complexity measure, we studied the fundamental different behaviors of linear regression model, tangent kernel (NTK) regression model and three-layer fully connected neural networks (NN). Our contribution is two-fold: we provided theoretical ground for using scaled optimism as a model predictive complexity measure; and we show empirically that NN with ReLUs behaves differently from kernel models under this measure. With resampling techniques, we can also compute the optimism for regression models with real data.