A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
This addresses the problem of performance degradation in diffusion models for AI researchers and practitioners, offering a practical mitigation strategy, though it is incremental as it builds on prior work on model collapse.
The paper tackles model collapse in diffusion models by identifying a transition from generalization to memorization during iterative training on synthetic data, driven by declining entropy, and proposes an entropy-based data selection strategy that significantly enhances visual quality and diversity to prevent collapse.
The widespread use of diffusion models has led to an abundance of AI-generated data, raising concerns about model collapse -- a phenomenon in which recursive iterations of training on synthetic data lead to performance degradation. Prior work primarily characterizes this collapse via variance shrinkage or distribution shift, but these perspectives miss practical manifestations of model collapse. This paper identifies a transition from generalization to memorization during model collapse in diffusion models, where models increasingly replicate training data instead of generating novel content during iterative training on synthetic samples. This transition is directly driven by the declining entropy of the synthetic training data produced in each training cycle, which serves as a clear indicator of model degradation. Motivated by this insight, we propose an entropy-based data selection strategy to mitigate the transition from generalization to memorization and alleviate model collapse. Empirical results show that our approach significantly enhances visual quality and diversity in recursive generation, effectively preventing collapse.