Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance
This work addresses the challenge of optimizing dataset choice for differentially private training, which is incremental as it builds on existing methods for noise reduction using public data.
The paper tackles the problem of selecting the most suitable public dataset for private machine learning by introducing an algorithm that measures gradient subspace distance between public and private data, with theoretical analysis showing excess risk scales with this distance and empirical results indicating model accuracy is monotone in the distance.
Differentially private stochastic gradient descent privatizes model training by injecting noise into each iteration, where the noise magnitude increases with the number of model parameters. Recent works suggest that we can reduce the noise by leveraging public data for private machine learning, by projecting gradients onto a subspace prescribed by the public data. However, given a choice of public datasets, it is not a priori clear which one may be most appropriate for the private task. We give an algorithm for selecting a public dataset by measuring a low-dimensional subspace distance between gradients of the public and private examples. We provide theoretical analysis demonstrating that the excess risk scales with this subspace distance. This distance is easy to compute and robust to modifications in the setting. Empirical evaluation shows that trained model accuracy is monotone in this distance.