Model Collapse Demystified: The Case of Regression
This work addresses model collapse, a critical issue for AI systems that rely on recursive training, but it is incremental as it focuses on a specific regression setting.
The authors tackled the problem of model collapse in high-dimensional regression, deriving analytic formulas that quantify performance degradation and proposing an adaptive regularization strategy to mitigate it, with theoretical results validated by experiments.
In the era of proliferation of large language and image generation models, the phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on data generated from previous generations of itself over time, its performance degrades until the model eventually becomes completely useless, i.e the model collapses. In this work, we study this phenomenon in the setting of high-dimensional regression and obtain analytic formulae which quantitatively outline this phenomenon in a broad range of regimes. In the special case of polynomial decaying spectral and source conditions, we obtain modified scaling laws which exhibit new crossover phenomena from fast to slow rates. We also propose a simple strategy based on adaptive regularization to mitigate model collapse. Our theoretical results are validated with experiments.