Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork
For practitioners of differentially private deep learning, this work offers a method to reduce the adverse effects of noise in gradient-based DP training, though it relies on public datasets and is domain-specific.
The authors propose a new framework for differentially private learning that avoids iterative noise injection by using a hypernetwork to map a private dataset to model parameters, reducing noise effects. In synthetic settings and LoRA fine-tuning of diffusion models, their approach achieves higher utility and lower FID compared to DP-SGD.
Differentially private (DP) training of neural networks is often hindered by the large amount of noise required by gradient-based methods such as DP-SGD, which repeatedly inject high-dimensional noise in parameter space throughout training. In this paper, we propose a new framework for DP learning that avoids iterative optimization in parameter space. Instead of updating the target model using privatized gradients, we employ a hypernetwork trained on public datasets to map a private dataset to the parameters of the target model. Specifically, each example is embedded into a low-dimensional representation, the embeddings are aggregated and perturbed to obtain a DP dataset embedding, and the hypernetwork generates the target model parameters from this noisy embedding. Because privacy noise is injected only once into a low-dimensional dataset representation, our approach can significantly reduce the adverse effect of noise. We theoretically show in a synthetic setting that, under a fixed privacy budget, models produced by our approach achieve higher utility than those trained with DP-SGD. Moreover, we apply our approach to LoRA fine-tuning of diffusion models and show that it achieves lower FID than LoRA models trained with DP-SGD and other public-data-guided methods.