The Geometry of Over-parameterized Regression and Adversarial Perturbations
This work provides theoretical insights into over-parameterization in machine learning, which is incremental but clarifies fundamental geometric aspects affecting model behavior.
The authors tackled the lack of a geometric interpretation for over-parameterized regression by developing a new feature-based perspective that explains the double-descent phenomenon and links adversarial perturbations to model bias. They demonstrated these ideas through analysis of minimal linear least squares models, including cases with linear or nonlinear basis functions.
Classical regression has a simple geometric description in terms of a projection of the training labels onto the column space of the design matrix. However, for over-parameterized models -- where the number of fit parameters is large enough to perfectly fit the training data -- this picture becomes uninformative. Here, we present an alternative geometric interpretation of regression that applies to both under- and over-parameterized models. Unlike the classical picture which takes place in the space of training labels, our new picture resides in the space of input features. This new feature-based perspective provides a natural geometric interpretation of the double-descent phenomenon in the context of bias and variance, explaining why it can occur even in the absence of label noise. Furthermore, we show that adversarial perturbations -- small perturbations to the input features that result in large changes in label values -- are a generic feature of biased models, arising from the underlying geometry. We demonstrate these ideas by analyzing three minimal models for over-parameterized linear least squares regression: without basis functions (input features equal model features) and with linear or nonlinear basis functions (two-layer neural networks with linear or nonlinear activation functions, respectively).