When Does More Regularization Imply Fewer Degrees of Freedom? Sufficient Conditions and Counter Examples from Lasso and Ridge Regression
This work addresses a foundational issue in statistical learning by revealing limitations of regularization, impacting researchers and practitioners who rely on these methods for model improvement.
The paper challenges the common belief that regularization always reduces degrees of freedom, showing counterexamples in lasso and ridge regression where it can increase degrees of freedom and training error, making regularization ineffective. It also identifies specific conditions, such as symmetric linear smoothers and convex constrained linear regression, where reduction in degrees of freedom is guaranteed.
Regularization aims to improve prediction performance of a given statistical modeling approach by moving to a second approach which achieves worse training error but is expected to have fewer degrees of freedom, i.e., better agreement between training and prediction error. We show here, however, that this expected behavior does not hold in general. In fact, counter examples are given that show regularization can increase the degrees of freedom in simple situations, including lasso and ridge regression, which are the most common regularization approaches in use. In such situations, the regularization increases both training error and degrees of freedom, and is thus inherently without merit. On the other hand, two important regularization scenarios are described where the expected reduction in degrees of freedom is indeed guaranteed: (a) all symmetric linear smoothers, and (b) linear regression versus convex constrained linear regression (as in the constrained variant of ridge regression and lasso).