Utility Theory of Synthetic Data Generation
This work addresses a theoretical gap for researchers and practitioners using synthetic data in machine learning, though it is incremental as it builds on existing empirical practices.
The paper tackles the lack of theoretical understanding of synthetic data utility by establishing a utility theory in a statistical learning framework, showing that synthetic data distributions do not need to closely mimic real data for comparable model generalization and consistent model ranking, with validation through experiments on non-parametric models and deep neural networks.
Synthetic data algorithms are widely employed in industries to generate artificial data for downstream learning tasks. While existing research primarily focuses on empirically evaluating utility of synthetic data, its theoretical understanding is largely lacking. This paper bridges the practice-theory gap by establishing relevant utility theory in a statistical learning framework. It considers two utility metrics: generalization and ranking of models trained on synthetic data. The former is defined as the generalization difference between models trained on synthetic and on real data. By deriving analytical bounds for this utility metric, we demonstrate that the synthetic feature distribution does not need to be similar as that of real data for ensuring comparable generalization of synthetic models, provided proper model specifications in downstream learning tasks. The latter utility metric studies the relative performance of models trained on synthetic data. In particular, we discover that the distribution of synthetic data is not necessarily similar as the real one to ensure consistent model comparison. Interestingly, consistent model comparison is still achievable even when synthetic responses are not well generated, as long as downstream models are separable by a generalization gap. Finally, extensive experiments on non-parametric models and deep neural networks have been conducted to validate these theoretical findings.