Multiple Descent: Design Your Own Generalization Curve
This work addresses the understanding of generalization behavior in machine learning, providing insights for researchers and practitioners, though it is incremental in extending known concepts.
The paper tackles the generalization loss of linear regression in variably parameterized models, showing that the generalization curve can have an arbitrary number of peaks with controllable locations, and reveals that U-shaped and double descent curves are not intrinsic but arise from data-algorithm interactions.
This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized. We show that the generalization curve can have an arbitrary number of peaks, and moreover, locations of those peaks can be explicitly controlled. Our results highlight the fact that both classical U-shaped generalization curve and the recently observed double descent curve are not intrinsic properties of the model family. Instead, their emergence is due to the interaction between the properties of the data and the inductive biases of learning algorithms.