A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasets
This work addresses a theoretical gap for researchers and practitioners using synthetic data, though it is incremental as it builds on existing empirical findings.
The authors tackled the lack of theoretical understanding for using multiple synthetic datasets in supervised learning by deriving bias-variance decompositions, which led to a rule of thumb for selecting the number of datasets and showed that multiple datasets often improve accuracy in practice.
Recent studies have highlighted the benefits of generating multiple synthetic datasets for supervised learning, from increased accuracy to more effective model selection and uncertainty estimation. These benefits have clear empirical support, but the theoretical understanding of them is currently very light. We seek to increase the theoretical understanding by deriving bias-variance decompositions for several settings of using multiple synthetic datasets, including differentially private synthetic data. Our theory yields a simple rule of thumb to select the appropriate number of synthetic datasets in the case of mean-squared error and Brier score. We investigate how our theory works in practice with several real datasets, downstream predictors and error metrics. As our theory predicts, multiple synthetic datasets often improve accuracy, while a single large synthetic dataset gives at best minimal improvement, showing that our insights are practically relevant.