Golden Ratio Weighting Prevents Model Collapse
This addresses a central challenge in generative model research by providing a theoretical framework to prevent performance degradation, though it appears incremental as it builds on known issues with model collapse.
The paper tackles model collapse in recursive generative model training by investigating optimal weighting schemes for mixing real and synthetic data, finding that the optimal weight for real data asymptotically follows a unified expression, sometimes equaling the reciprocal of the golden ratio, and validates this on simulated and real datasets.
Recent studies identified an intriguing phenomenon in recursive generative model training known as model collapse, where models trained on data generated by previous models exhibit severe performance degradation. Addressing this issue and developing more effective training strategies have become central challenges in generative model research. In this paper, we investigate this phenomenon within a novel framework, where generative models are iteratively trained on a combination of newly collected real data and synthetic data from the previous training step. To develop an optimal training strategy for integrating real and synthetic data, we evaluate the performance of a weighted training scheme in various scenarios, including Gaussian distribution estimation, generalized linear models, and nonparametric estimation. We theoretically characterize the impact of the mixing proportion and weighting scheme of synthetic data on the final model's performance. Our key finding is that, across different settings, the optimal weighting scheme under different proportions of synthetic data asymptotically follows a unified expression, revealing a fundamental trade-off between leveraging synthetic data and model performance. In some cases, the optimal weight assigned to real data corresponds to the reciprocal of the golden ratio. Finally, we validate our theoretical results on extensive simulated datasets and a real tabular dataset.