On Regularization via Early Stopping for Least Squares Regression
This work addresses the fundamental problem of understanding early stopping's effect on generalization in machine learning, offering theoretical insights for practitioners in regression tasks.
The paper analyzes the dynamics of gradient descent for linear regression, showing that early stopping yields a solution equivalent to a generalized ridge regularization and is beneficial for generic data with arbitrary spectrum, providing an estimate for the optimal stopping time.
A fundamental problem in machine learning is understanding the effect of early stopping on the parameters obtained and the generalization capabilities of the model. Even for linear models, the effect is not fully understood for arbitrary learning rates and data. In this paper, we analyze the dynamics of discrete full batch gradient descent for linear regression. With minimal assumptions, we characterize the trajectory of the parameters and the expected excess risk. Using this characterization, we show that when training with a learning rate schedule $η_k$, and a finite time horizon $T$, the early stopped solution $β_T$ is equivalent to the minimum norm solution for a generalized ridge regularized problem. We also prove that early stopping is beneficial for generic data with arbitrary spectrum and for a wide variety of learning rate schedules. We provide an estimate for the optimal stopping time and empirically demonstrate the accuracy of our estimate.