No Free Lunch for Synthetic Images under Data Scarcity Conditions
For researchers and practitioners using synthetic data in privacy-sensitive domains, this work highlights the need for multidimensional evaluation and shows that model choice matters under privacy constraints.
This study evaluates the trade-offs between fidelity, privacy, and utility in synthetic data generation under data scarcity, finding that GAN and DDPM are more robust than VAE when differential privacy is applied.
This study investigates the trade-offs between fidelity, privacy, and utility in synthetic data generation under conditions of data scarcity and privacy sensitivity. We propose an evaluation framework that jointly assesses these three dimensions and apply it to three widely used generative models, VAE, GAN, and DDPM. The evaluation spans three image datasets, MNIST, OCTMNIST, and OrganAMNIST, encompassing both general-purpose and medical imaging domains. Notable differences arise between the three models in their behaviour when differential privacy mechanisms are introduced during training. GAN and DDPM demonstrate greater robustness, maintaining higher fidelity and downstream utility across a range of noise levels, while VAE degrades more rapidly as privacy constraints increase. This study highlights the importance of a multidimensional evaluation of deep generative models, also noting that their behaviour significantly differs when privacy techniques are applied.