Measuring the quality of Synthetic data for use in competitions
This work addresses the challenge of enabling data sharing for ML research while protecting privacy, though it is incremental as it focuses on a specific evaluation criterion rather than introducing new synthetic data methods.
The paper tackles the problem of evaluating synthetic data quality for machine learning competitions by proposing that synthetic data should preserve the relative performance ranking of algorithms compared to the original dataset, aiming to ensure utility without compromising privacy.
Machine learning has the potential to assist many communities in using the large datasets that are becoming more and more available. Unfortunately, much of that potential is not being realized because it would require sharing data in a way that compromises privacy. In order to overcome this hurdle, several methods have been proposed that generate synthetic data while preserving the privacy of the real data. In this paper we consider a key characteristic that synthetic data should have in order to be useful for machine learning researchers - the relative performance of two algorithms (trained and tested) on the synthetic dataset should be the same as their relative performance (when trained and tested) on the original dataset.