Beyond Real Data: Synthetic Data through the Lens of Regularization
This work addresses the challenge of data scarcity and distributional mismatches for machine learning practitioners, offering a theoretical and practical guide for optimizing synthetic data use, though it is incremental in extending existing regularization and stability concepts.
The paper tackles the problem of balancing synthetic and real data to improve generalization, deriving a theoretical framework that predicts an optimal synthetic-to-real data ratio minimizing test error, with empirical validation on CIFAR-10 and a clinical MRI dataset showing U-shaped error behavior.
Synthetic data can improve generalization when real data is scarce, but excessive reliance may introduce distributional mismatches that degrade performance. In this paper, we present a learning-theoretic framework to quantify the trade-off between synthetic and real data. Our approach leverages algorithmic stability to derive generalization error bounds, characterizing the optimal synthetic-to-real data ratio that minimizes expected test error as a function of the Wasserstein distance between the real and synthetic distributions. We motivate our framework in the setting of kernel ridge regression with mixed data, offering a detailed analysis that may be of independent interest. Our theory predicts the existence of an optimal ratio, leading to a U-shaped behavior of test error with respect to the proportion of synthetic data. Empirically, we validate this prediction on CIFAR-10 and a clinical brain MRI dataset. Our theory extends to the important scenario of domain adaptation, showing that carefully blending synthetic target data with limited source data can mitigate domain shift and enhance generalization. We conclude with practical guidance for applying our results to both in-domain and out-of-domain scenarios.