A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops
This addresses the critical issue of data scarcity for training large generative models, providing theoretical insights to prevent degradation in self-training scenarios.
The paper tackles the problem of model collapse in self-consuming training loops (STLs) for generative models, revealing through theoretical analysis that model architecture and the proportion of real to synthetic data influence success, with findings showing that a constant proportion of real data ensures convergence in transformers.
High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted. Consequently, models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs). However, the empirical results have been strikingly inconsistent: some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding to explain this discrepancy. This paper introduces the intriguing notion of recursive stability and presents the first theoretical generalization analysis, revealing how both model architecture and the proportion between real and synthetic data influence the success of STLs. We further extend this analysis to transformers in in-context learning, showing that even a constant-sized proportion of real data ensures convergence, while also providing insights into optimal synthetic data sizing.